[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:18:44 dist-test-slave-2w3w krb5kdc[3628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:18:44 dist-test-slave-2w3w krb5kdc[3628](info): set up 2 sockets
Aug 09 06:18:44 dist-test-slave-2w3w krb5kdc[3628](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 06:18:45.575209 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:18:45 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:18:45.760983 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33717
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:33717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:46.216184 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:46.216580 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:46.216781 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:46.225559 3642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:18:46.225767 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:46.225940 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:46.226117 3642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:18:46.226258 3642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:18:46.238795 3642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:33717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:46.240273 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:46.241658 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:46.258010 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:46.258317 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:46.258689 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:46.260161 3642 server_base.cc:1034] running on GCE node
I20260809 06:18:46.261175 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:46.262754 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:46.263983 3642 hybrid_clock.cc:648] HybridClock initialized: now 1786256326263945 us; error 71 us; skew 500 ppm
I20260809 06:18:46.264341 3642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:46 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:46.267616 3642 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:18:46.269568 3642 webserver.cc:460] Webserver started at http://127.3.137.126:45533/ using document root <none> and password file <none>
I20260809 06:18:46.270376 3642 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:46.270588 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:46.271056 3642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:46.274595 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "6783796bc5a2422cadc86aae0d09d563"
format_stamp: "Formatted at 2026-08-09 06:18:46 on dist-test-slave-2w3w"
server_key: "71e484f48a74bdb9f819b7cf1cc0fe4a"
server_key_iv: "e76f7ff6397dd9fbac0d8d64d9e645ac"
server_key_version: "encryptionkey@0"
I20260809 06:18:46.275728 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "6783796bc5a2422cadc86aae0d09d563"
format_stamp: "Formatted at 2026-08-09 06:18:46 on dist-test-slave-2w3w"
server_key: "71e484f48a74bdb9f819b7cf1cc0fe4a"
server_key_iv: "e76f7ff6397dd9fbac0d8d64d9e645ac"
server_key_version: "encryptionkey@0"
I20260809 06:18:46.282091 3642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260809 06:18:46.285931 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:46.287282 3642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:18:46.287456 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "6783796bc5a2422cadc86aae0d09d563"
format_stamp: "Formatted at 2026-08-09 06:18:46 on dist-test-slave-2w3w"
server_key: "71e484f48a74bdb9f819b7cf1cc0fe4a"
server_key_iv: "e76f7ff6397dd9fbac0d8d64d9e645ac"
server_key_version: "encryptionkey@0"
I20260809 06:18:46.287719 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:46.312331 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:46.317545 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:46.318084 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:46.348623 3642 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:33717
I20260809 06:18:46.348603 3717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:33717 every 8 connection(s)
I20260809 06:18:46.350339 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:18:46.352231 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 3642
I20260809 06:18:46.352662 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:18:46.355002 3621 external_mini_cluster.cc:1428] Setting key 5bceaedea05e9793d2339de536ead460
I20260809 06:18:46.356019 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:46.369758 3718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563: Bootstrap starting.
I20260809 06:18:46.374003 3718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:46.375209 3718 log.cc:824] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:46.378139 3718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563: No bootstrap required, opened a new log
I20260809 06:18:46.386391 3718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } }
Aug 09 06:18:46 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:46.386909 3718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:46.387042 3718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6783796bc5a2422cadc86aae0d09d563, State: Initialized, Role: FOLLOWER
I20260809 06:18:46.387727 3718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } }
I20260809 06:18:46.388043 3718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:18:46.388202 3718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:18:46.388382 3718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:46.391471 3718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } }
I20260809 06:18:46.392055 3718 leader_election.cc:302] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6783796bc5a2422cadc86aae0d09d563; no voters:
I20260809 06:18:46.393078 3718 leader_election.cc:288] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:18:46.393388 3723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:46.395613 3723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [term 1 LEADER]: Becoming Leader. State: Replica: 6783796bc5a2422cadc86aae0d09d563, State: Running, Role: LEADER
I20260809 06:18:46.396014 3718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:18:46.396585 3723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } }
I20260809 06:18:46.400970 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:46.359571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.3.137.126:33717)
0809 06:18:46.361212 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:46.361231 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:46.362484 (+ 1253us) server_negotiation.cc:407] Connection header received
0809 06:18:46.373302 (+ 10818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:46.373322 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:46.373668 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:46.374047 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:46.377364 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:46.379078 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:46.380965 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:46.381751 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:46.382010 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:46.388634 (+ 6624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:46.388642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:46.388654 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:46.388697 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:46.392817 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:46.394032 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:46.394044 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:46.394055 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:46.394368 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:46.394997 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:46.395003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:46.395006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:46.395839 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:46.396309 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:46.397865 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0809 06:18:46.398461 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":902,"threads_started":1}
I20260809 06:18:46.404848 3725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6783796bc5a2422cadc86aae0d09d563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } } }
I20260809 06:18:46.405267 3725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:46.407397 3727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6783796bc5a2422cadc86aae0d09d563. Latest consensus state: current_term: 1 leader_uuid: "6783796bc5a2422cadc86aae0d09d563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6783796bc5a2422cadc86aae0d09d563" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33717 } } }
I20260809 06:18:46.407860 3727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563 [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:46.407984 3731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:18:46.418838 3731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:18:46.429827 3731 catalog_manager.cc:1347] Generated new cluster ID: 79889d5129d94d95bc2cc4371df0ef7c
I20260809 06:18:46.429993 3731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:18:46.444903 3731 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:18:46.446204 3731 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:18:46.452836 3731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6783796bc5a2422cadc86aae0d09d563: Generated new TSK 0
I20260809 06:18:46.453863 3731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:18:46.511641 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:46.969228 3748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:46.969507 3748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:46.969660 3748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:46.978078 3748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:46.978214 3748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:46.978390 3748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:18:46.989784 3748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:46.991328 3748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:46.992870 3748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:47.009037 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:47.009646 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:47.009884 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:47.011214 3748 server_base.cc:1034] running on GCE node
I20260809 06:18:47.011916 3748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:47.012916 3748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:47.014125 3748 hybrid_clock.cc:648] HybridClock initialized: now 1786256327014056 us; error 55 us; skew 500 ppm
I20260809 06:18:47.014467 3748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:47 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:47.019032 3748 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:18:47.020675 3748 webserver.cc:460] Webserver started at http://127.3.137.65:36227/ using document root <none> and password file <none>
I20260809 06:18:47.021406 3748 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:47.021541 3748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:47.021883 3748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:47.025267 3748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "9f6c9c87b68b4264bcb2e27fd3beba2b"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "fb3305eca05c399927e0b928277ccf4a"
server_key_iv: "f8079f649139db622dd7c8a90741a632"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.026196 3748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "9f6c9c87b68b4264bcb2e27fd3beba2b"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "fb3305eca05c399927e0b928277ccf4a"
server_key_iv: "f8079f649139db622dd7c8a90741a632"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.033447 3748 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260809 06:18:47.038411 3764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:47.039755 3748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:18:47.039927 3748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "9f6c9c87b68b4264bcb2e27fd3beba2b"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "fb3305eca05c399927e0b928277ccf4a"
server_key_iv: "f8079f649139db622dd7c8a90741a632"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.040130 3748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:47.055856 3748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:47.060222 3748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:47.060545 3748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:47.061520 3748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:47.063025 3748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:47.063165 3748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 06:18:47.063380 3748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:47.063503 3748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:47.124543 3748 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:44859
I20260809 06:18:47.124567 3886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:44859 every 8 connection(s)
I20260809 06:18:47.126263 3748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:18:47.134236 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 3748
I20260809 06:18:47.134581 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:18:47.135357 3621 external_mini_cluster.cc:1428] Setting key d1192fc68a7613b30dca93020d56e560
Aug 09 06:18:47 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:47.150619 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:47.129249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48379 (local address 127.3.137.126:33717)
0809 06:18:47.129800 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:47.129805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:47.130613 (+ 808us) server_negotiation.cc:407] Connection header received
0809 06:18:47.132532 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:47.132538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:47.132627 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:47.132744 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:47.134986 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.135958 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.138355 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.138780 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.138891 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:47.144115 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:47.144120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:47.144123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:47.144153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:47.147331 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:47.148029 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:47.148035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:47.148038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:47.148209 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:47.148642 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:47.148648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:47.148651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:47.148987 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:47.149143 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:47.149919 (+ 776us) server_negotiation.cc:299] Negotiation successful
0809 06:18:47.150143 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":317,"threads_started":1}
I20260809 06:18:47.151576 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:47.129698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33717 (local address 127.3.137.65:48379)
0809 06:18:47.130460 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:47.130496 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:47.132265 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:47.132980 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:47.132988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:47.133562 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:47.134770 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:47.134799 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.136162 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.136166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:47.138176 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:47.138191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.139093 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.139105 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:47.139544 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:47.140366 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:47.140398 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:47.143895 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:47.147468 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:47.147479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:47.147495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:47.147913 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:47.148332 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:47.148337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:47.148340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:47.148530 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:47.149117 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:47.149125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:47.149454 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 06:18:47.149719 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 06:18:47.150265 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":264,"threads_started":1}
I20260809 06:18:47.153360 3887 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33717
I20260809 06:18:47.153659 3887 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:47.154567 3887 heartbeater.cc:511] Master 127.3.137.126:33717 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:18:47.157245 3678 ts_manager.cc:194] Registered new tserver with Master: 9f6c9c87b68b4264bcb2e27fd3beba2b (127.3.137.65:44859)
I20260809 06:18:47.159133 3678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:48379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:18:47.183969 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:47.532867 3893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:47.533175 3893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:47.533284 3893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:47.541172 3893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:47.541285 3893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:47.541401 3893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:18:47.552444 3893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:47.553665 3893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:47.555042 3893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:47.568857 3898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:47.569033 3901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:47.569063 3899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:47.570035 3893 server_base.cc:1034] running on GCE node
I20260809 06:18:47.570662 3893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:47.571619 3893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:47.572839 3893 hybrid_clock.cc:648] HybridClock initialized: now 1786256327572794 us; error 36 us; skew 500 ppm
I20260809 06:18:47.573170 3893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:47 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:47.576476 3893 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:18:47.578074 3893 webserver.cc:460] Webserver started at http://127.3.137.66:38245/ using document root <none> and password file <none>
I20260809 06:18:47.578740 3893 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:47.578841 3893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:47.579193 3893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:47.582248 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "f42043f5d5b84f2c8d31a41d082f80fc"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "59257cff03e6f391a7d4c8034d12bce1"
server_key_iv: "d79d593573398360c37575f9cae1cddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.583091 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "f42043f5d5b84f2c8d31a41d082f80fc"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "59257cff03e6f391a7d4c8034d12bce1"
server_key_iv: "d79d593573398360c37575f9cae1cddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.588476 3893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260809 06:18:47.592218 3908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:47.593559 3893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:18:47.593734 3893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "f42043f5d5b84f2c8d31a41d082f80fc"
format_stamp: "Formatted at 2026-08-09 06:18:47 on dist-test-slave-2w3w"
server_key: "59257cff03e6f391a7d4c8034d12bce1"
server_key_iv: "d79d593573398360c37575f9cae1cddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:47.593946 3893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:47.624115 3893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:47.628592 3893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:47.628921 3893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:47.629914 3893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:47.631281 3893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:47.631382 3893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:47.631531 3893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:47.631613 3893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:47.666515 3893 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:38977
I20260809 06:18:47.666538 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:38977 every 8 connection(s)
I20260809 06:18:47.668020 3893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:18:47.671036 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 3893
I20260809 06:18:47.671234 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:18:47.671648 3621 external_mini_cluster.cc:1428] Setting key 730f56d529ccd9bb8dfee229673896cb
Aug 09 06:18:47 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:47.693176 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:47.670747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58015 (local address 127.3.137.126:33717)
0809 06:18:47.671293 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:47.671299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:47.672341 (+ 1042us) server_negotiation.cc:407] Connection header received
0809 06:18:47.674781 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:47.674785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:47.674849 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:47.674941 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:47.677917 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.678666 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.681508 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.681951 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.682096 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:47.686461 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:47.686466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:47.686469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:47.686502 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:47.689096 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:47.690444 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:47.690448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:47.690450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:47.690575 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:47.691148 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:47.691151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:47.691154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:47.691423 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:47.691541 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:47.692610 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0809 06:18:47.692858 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260809 06:18:47.694587 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:47.671326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33717 (local address 127.3.137.66:58015)
0809 06:18:47.672164 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:47.672209 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:47.674426 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:47.675288 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:47.675303 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:47.676449 (+ 1146us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:47.677648 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:47.677686 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.678884 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.678892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:47.681356 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:47.681373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:47.682208 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:47.682217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:47.682442 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:47.683266 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:47.683298 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:47.686202 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:47.689263 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:47.689276 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:47.689298 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:47.690278 (+ 980us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:47.690715 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:47.690723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:47.690730 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:47.691028 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:47.691557 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:47.691568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:47.692070 (+ 502us) client_negotiation.cc:769] Sending connection context
0809 06:18:47.692474 (+ 404us) client_negotiation.cc:240] Negotiation successful
0809 06:18:47.693096 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:18:47.696188 4028 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33717
I20260809 06:18:47.696509 4028 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:47.697351 4028 heartbeater.cc:511] Master 127.3.137.126:33717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
I20260809 06:18:47.698940 3678 ts_manager.cc:194] Registered new tserver with Master: f42043f5d5b84f2c8d31a41d082f80fc (127.3.137.66:38977)
I20260809 06:18:47.699906 3678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:58015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:18:47.722579 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:48.049233 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:48.049507 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:48.049687 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:48.056886 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:48.056999 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:48.057179 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:18:48.066519 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33717
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:48.067746 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:48.069020 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:48.081736 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:48.081776 4041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:48.081600 4043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:48.082850 4034 server_base.cc:1034] running on GCE node
I20260809 06:18:48.083491 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:48.084344 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:48.085556 4034 hybrid_clock.cc:648] HybridClock initialized: now 1786256328085504 us; error 43 us; skew 500 ppm
I20260809 06:18:48.085906 4034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:48.089105 4034 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:18:48.090590 4034 webserver.cc:460] Webserver started at http://127.3.137.67:37979/ using document root <none> and password file <none>
I20260809 06:18:48.091321 4034 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:48.091473 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:48.091876 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:48.095139 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "6e550762299a4fffa610d8e008eb4064"
format_stamp: "Formatted at 2026-08-09 06:18:48 on dist-test-slave-2w3w"
server_key: "39a7766044657e61e8e2c9108653f2d0"
server_key_iv: "9d297ed8d94f1fe91814981f3cbd7693"
server_key_version: "encryptionkey@0"
I20260809 06:18:48.096056 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "6e550762299a4fffa610d8e008eb4064"
format_stamp: "Formatted at 2026-08-09 06:18:48 on dist-test-slave-2w3w"
server_key: "39a7766044657e61e8e2c9108653f2d0"
server_key_iv: "9d297ed8d94f1fe91814981f3cbd7693"
server_key_version: "encryptionkey@0"
I20260809 06:18:48.101765 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:18:48.105417 4051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:48.106860 4034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:18:48.107097 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "6e550762299a4fffa610d8e008eb4064"
format_stamp: "Formatted at 2026-08-09 06:18:48 on dist-test-slave-2w3w"
server_key: "39a7766044657e61e8e2c9108653f2d0"
server_key_iv: "9d297ed8d94f1fe91814981f3cbd7693"
server_key_version: "encryptionkey@0"
I20260809 06:18:48.107348 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:48.125869 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:48.130612 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:48.130985 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:48.132016 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:48.133502 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:48.133615 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:48.133777 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:48.133857 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:48.162772 3887 heartbeater.cc:503] Master 127.3.137.126:33717 was elected leader, sending a full tablet report...
I20260809 06:18:48.173400 4034 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:35603
I20260809 06:18:48.173420 4172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:35603 every 8 connection(s)
I20260809 06:18:48.174739 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:18:48.181582 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4034
I20260809 06:18:48.181843 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:18:48.182298 3621 external_mini_cluster.cc:1428] Setting key 138d5c4a6e4f544bc2c8e33aac79d8fa
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:48.195465 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.177375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:53831 (local address 127.3.137.126:33717)
0809 06:18:48.177553 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.177557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.178907 (+ 1350us) server_negotiation.cc:407] Connection header received
0809 06:18:48.181272 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.181281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.181396 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.181521 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.183356 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.184285 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.186161 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.186560 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.186664 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.189959 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.189964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.189967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.189999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.192816 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.193487 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.193491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.193493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.193608 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.193953 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.193956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.193958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.194192 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.194324 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.195052 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.195240 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:18:48.196518 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.177930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33717 (local address 127.3.137.67:53831)
0809 06:18:48.178693 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:48.178791 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:48.180980 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:48.181778 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:48.181789 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:48.182479 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:48.183174 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:48.183200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.184443 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.184450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:48.186017 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:48.186033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.186751 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.186756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:48.186945 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.187630 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:48.187656 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:48.189777 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:48.192943 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:48.192958 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:48.192975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:48.193373 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:48.193691 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:48.193694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:48.193696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:48.193862 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:48.194296 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:48.194301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:48.194580 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 06:18:48.194821 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 06:18:48.195374 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":353,"threads_started":1}
I20260809 06:18:48.198143 4173 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33717
I20260809 06:18:48.198472 4173 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:48.199255 4173 heartbeater.cc:511] Master 127.3.137.126:33717 requested a full tablet report, sending...
I20260809 06:18:48.200512 3678 ts_manager.cc:194] Registered new tserver with Master: 6e550762299a4fffa610d8e008eb4064 (127.3.137.67:35603)
I20260809 06:18:48.201287 3678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:53831
I20260809 06:18:48.210366 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:48.260782 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.244798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.3.137.65:44859)
0809 06:18:48.245534 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.245543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.245578 (+ 35us) server_negotiation.cc:407] Connection header received
0809 06:18:48.245669 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.245676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.245892 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.246068 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.247602 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.248605 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.249877 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.250449 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.250882 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.255045 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.255056 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.255070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.255104 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.257874 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.258572 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.258586 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.258588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.258715 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.259175 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.259180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.259183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.259519 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.259841 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.260145 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.260423 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":383,"threads_started":1}
W20260809 06:18:48.262049 3794 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38176
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:48.308563 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.294682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.3.137.65:44859)
0809 06:18:48.294936 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.294945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.295304 (+ 359us) server_negotiation.cc:407] Connection header received
0809 06:18:48.295548 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.295552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.295619 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.295722 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.297227 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.297998 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.299269 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.299678 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.299806 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.303576 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.303580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.303584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.303628 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.306010 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.306672 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.306677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.306679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.306797 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.307213 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.307222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.307225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.307460 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.307711 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.308010 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.308250 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:18:48.309988 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 3748
I20260809 06:18:48.321689 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.311072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.3.137.65:44859)
0809 06:18:48.311212 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.311215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.311596 (+ 381us) server_negotiation.cc:407] Connection header received
0809 06:18:48.311693 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.311696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.311747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.311819 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.312874 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.313559 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.314757 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.315270 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.315354 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.317442 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.317446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.317449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.317479 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.319776 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.320323 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.320326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.320329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.320443 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.320719 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.320722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.320724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.320931 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.321105 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.321282 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.321419 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 06:18:48.322438 3881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:48.405333 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 3748
I20260809 06:18:48.426753 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 3893
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:18:48.442312 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.427808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.3.137.66:38977)
0809 06:18:48.428398 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.428405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.428431 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:18:48.428505 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.428512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.428719 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.428853 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.429984 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.431130 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.432321 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.432858 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.432956 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.437073 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.437083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.437096 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.437127 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.439940 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.440528 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.440534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.440536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.440674 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.441011 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.441014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.441016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.441313 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.441573 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.441795 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.441995 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":337,"threads_started":1}
I20260809 06:18:48.443605 4023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:48.525367 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 3893
I20260809 06:18:48.546259 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4034
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:18:48.562592 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:48.547538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.3.137.67:35603)
0809 06:18:48.548334 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:48.548344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:48.548469 (+ 125us) server_negotiation.cc:407] Connection header received
0809 06:18:48.548652 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:48.548659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:48.548865 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:48.548998 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:48.550217 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.551371 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.552657 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:48.553189 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:48.553281 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:48.557376 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:48.557386 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:48.557400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:48.557431 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:48.560239 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.560816 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.560823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.560825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.560946 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:48.561255 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:48.561258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:48.561260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:48.561588 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:48.561809 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:48.562071 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 06:18:48.562276 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":405,"threads_started":1}
I20260809 06:18:48.563912 4167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:48.646569 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4034
I20260809 06:18:48.668280 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 3642
I20260809 06:18:48.669445 3712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:48.772635 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 3642
2026-08-09T06:18:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4374 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[4197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[4197](info): set up 2 sockets
Aug 09 06:18:48 dist-test-slave-2w3w krb5kdc[4197](info): commencing operation
krb5kdc: starting...
W20260809 06:18:49.896396 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:18:50 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:18:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:18:50.083076 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:34697
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:50.400529 4212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:50.400852 4212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:50.401007 4212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:50.409046 4212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:18:50.409165 4212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:50.409271 4212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:50.409373 4212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:18:50.409446 4212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:18:50.419849 4212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:50.421236 4212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:50.422531 4212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:50.434840 4221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:50.434927 4219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:50.434949 4218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:50.435864 4212 server_base.cc:1034] running on GCE node
I20260809 06:18:50.436616 4212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:50.438158 4212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:50.439354 4212 hybrid_clock.cc:648] HybridClock initialized: now 1786256330439203 us; error 146 us; skew 500 ppm
I20260809 06:18:50.439731 4212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:50 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:50.446501 4212 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:18:50.448050 4212 webserver.cc:460] Webserver started at http://127.3.137.126:36689/ using document root <none> and password file <none>
I20260809 06:18:50.448689 4212 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:50.448796 4212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:50.449047 4212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:50.452028 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "128fca09f4c94f5a87b07db458a7ffcc"
format_stamp: "Formatted at 2026-08-09 06:18:50 on dist-test-slave-2w3w"
server_key: "313680b2c116cf068cad35fcc2d207ad"
server_key_iv: "de783a2e075e576ac11df123cb83bbc1"
server_key_version: "encryptionkey@0"
I20260809 06:18:50.452721 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "128fca09f4c94f5a87b07db458a7ffcc"
format_stamp: "Formatted at 2026-08-09 06:18:50 on dist-test-slave-2w3w"
server_key: "313680b2c116cf068cad35fcc2d207ad"
server_key_iv: "de783a2e075e576ac11df123cb83bbc1"
server_key_version: "encryptionkey@0"
I20260809 06:18:50.458038 4212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:18:50.461546 4229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:50.463097 4212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:18:50.463311 4212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "128fca09f4c94f5a87b07db458a7ffcc"
format_stamp: "Formatted at 2026-08-09 06:18:50 on dist-test-slave-2w3w"
server_key: "313680b2c116cf068cad35fcc2d207ad"
server_key_iv: "de783a2e075e576ac11df123cb83bbc1"
server_key_version: "encryptionkey@0"
I20260809 06:18:50.463574 4212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:50.481452 4212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:50.486191 4212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:50.486516 4212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:50.506791 4212 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:34697
I20260809 06:18:50.506824 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:34697 every 8 connection(s)
I20260809 06:18:50.508219 4212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:18:50.509258 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4212
I20260809 06:18:50.509502 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:18:50.510085 3621 external_mini_cluster.cc:1428] Setting key 1b1caa98eb3ce52ca6871fd6e8f82d87
I20260809 06:18:50.512771 4284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:18:50 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:50.526518 4284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc: Bootstrap starting.
I20260809 06:18:50.531181 4284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:50.532579 4284 log.cc:824] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:50.535866 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:50.511992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.3.137.126:34697)
0809 06:18:50.512856 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:50.512871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:50.512922 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:18:50.515380 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:50.515413 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:50.515957 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:50.516433 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:50.517766 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:50.520123 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:50.521342 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:50.522437 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:50.522631 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:50.526563 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:50.526576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:50.526599 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:50.526655 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:50.530890 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:50.531736 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:50.531745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:50.531754 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:50.531974 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:50.532344 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:50.532351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:50.532357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:50.533088 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:50.533447 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:50.533974 (+ 527us) server_negotiation.cc:299] Negotiation successful
0809 06:18:50.534443 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":376,"threads_started":1}
I20260809 06:18:50.536360 4284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc: No bootstrap required, opened a new log
I20260809 06:18:50.544795 4284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } }
I20260809 06:18:50.545290 4284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:50.545416 4284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128fca09f4c94f5a87b07db458a7ffcc, State: Initialized, Role: FOLLOWER
I20260809 06:18:50.546065 4284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } }
I20260809 06:18:50.546340 4284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:18:50.546504 4284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:18:50.546708 4284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:50.549306 4284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } }
I20260809 06:18:50.549831 4284 leader_election.cc:302] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128fca09f4c94f5a87b07db458a7ffcc; no voters:
I20260809 06:18:50.550477 4284 leader_election.cc:288] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:18:50.550631 4290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:50.551342 4290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [term 1 LEADER]: Becoming Leader. State: Replica: 128fca09f4c94f5a87b07db458a7ffcc, State: Running, Role: LEADER
I20260809 06:18:50.551862 4290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } }
I20260809 06:18:50.552645 4284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:18:50.554877 4292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128fca09f4c94f5a87b07db458a7ffcc. Latest consensus state: current_term: 1 leader_uuid: "128fca09f4c94f5a87b07db458a7ffcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } } }
I20260809 06:18:50.555418 4292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:50.555418 4291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128fca09f4c94f5a87b07db458a7ffcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128fca09f4c94f5a87b07db458a7ffcc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34697 } } }
I20260809 06:18:50.561548 4291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:50.562443 4296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:18:50.568363 4296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:18:50.577437 4296 catalog_manager.cc:1347] Generated new cluster ID: 0de401648a944046a7d7653f0fa4a3a5
I20260809 06:18:50.577556 4296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:18:50.588892 4296 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:18:50.590322 4296 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:18:50.599538 4296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128fca09f4c94f5a87b07db458a7ffcc: Generated new TSK 0
I20260809 06:18:50.600447 4296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:18:50.651575 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:50.974102 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:50.974445 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:50.974629 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:50.981959 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:50.982126 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:50.982326 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:18:50.992362 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:50.993705 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:50.995142 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:51.008528 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:51.008558 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:51.008656 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:51.010674 4312 server_base.cc:1034] running on GCE node
I20260809 06:18:51.011384 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:51.012372 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:51.013603 4312 hybrid_clock.cc:648] HybridClock initialized: now 1786256331013544 us; error 59 us; skew 500 ppm
I20260809 06:18:51.013922 4312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:51 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:51.017184 4312 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:18:51.018920 4312 webserver.cc:460] Webserver started at http://127.3.137.65:43557/ using document root <none> and password file <none>
I20260809 06:18:51.019707 4312 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:51.019865 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:51.020183 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:51.023097 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "d3a9279bcb8b4d7989f5525daaab3262"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "bc899d91c74ce58952400825c2cb10cd"
server_key_iv: "98faff4bb25cd605f09e349add8adb84"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.023934 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a9279bcb8b4d7989f5525daaab3262"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "bc899d91c74ce58952400825c2cb10cd"
server_key_iv: "98faff4bb25cd605f09e349add8adb84"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.029125 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:18:51.032799 4328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:51.034138 4312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:18:51.034344 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "d3a9279bcb8b4d7989f5525daaab3262"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "bc899d91c74ce58952400825c2cb10cd"
server_key_iv: "98faff4bb25cd605f09e349add8adb84"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.034559 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:51.061698 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:51.066196 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:51.066505 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:51.067605 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:51.068881 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:51.068987 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:51.069137 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:51.069211 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:51.102789 4312 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:44005
I20260809 06:18:51.102843 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:44005 every 8 connection(s)
I20260809 06:18:51.104082 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:18:51.110661 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4312
I20260809 06:18:51.110885 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:18:51.111506 3621 external_mini_cluster.cc:1428] Setting key 96a3b7bbed66cfa3786a220fe8e13ae7
Aug 09 06:18:51 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:51.123747 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:51.106636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:35089 (local address 127.3.137.126:34697)
0809 06:18:51.107151 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:51.107156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:51.107992 (+ 836us) server_negotiation.cc:407] Connection header received
0809 06:18:51.109811 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:51.109816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:51.109905 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:51.110028 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:51.111565 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.112321 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.113853 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.114236 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.114384 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:51.117958 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:51.117963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:51.117966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:51.117997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:51.120661 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:51.121413 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:51.121419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:51.121427 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:51.121539 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:51.121972 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:51.121978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:51.121981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:51.122235 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:51.122397 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:51.123114 (+ 717us) server_negotiation.cc:299] Negotiation successful
0809 06:18:51.123348 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
I20260809 06:18:51.124666 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:51.107131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34697 (local address 127.3.137.65:35089)
0809 06:18:51.107831 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:51.107868 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:51.109544 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:51.110257 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:51.110265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:51.110772 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:51.111398 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:51.111415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.112484 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.112495 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:51.113696 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:51.113708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.114467 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.114475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:51.114723 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:51.115419 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:51.115441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:51.117730 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:51.120842 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:51.120851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:51.120868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:51.121284 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:51.121652 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:51.121658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:51.121662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:51.121850 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:51.122377 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:51.122384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:51.122705 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 06:18:51.122999 (+ 294us) client_negotiation.cc:240] Negotiation successful
0809 06:18:51.123531 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
I20260809 06:18:51.126327 4451 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34697
I20260809 06:18:51.126703 4451 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:51.127638 4451 heartbeater.cc:511] Master 127.3.137.126:34697 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:18:51.130159 4247 ts_manager.cc:194] Registered new tserver with Master: d3a9279bcb8b4d7989f5525daaab3262 (127.3.137.65:44005)
I20260809 06:18:51.131673 4247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:35089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:18:51.154830 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:51.471999 4458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:51.472299 4458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:51.472453 4458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:51.479574 4458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:51.479710 4458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:51.479962 4458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:18:51.489666 4458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:51.491168 4458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:51.492465 4458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:51.505192 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:51.505363 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:51.505332 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:51.506418 4458 server_base.cc:1034] running on GCE node
I20260809 06:18:51.507134 4458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:51.508291 4458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:51.509536 4458 hybrid_clock.cc:648] HybridClock initialized: now 1786256331509477 us; error 53 us; skew 500 ppm
I20260809 06:18:51.509925 4458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:51 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:51.513814 4458 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:18:51.515574 4458 webserver.cc:460] Webserver started at http://127.3.137.66:46169/ using document root <none> and password file <none>
I20260809 06:18:51.516268 4458 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:51.516422 4458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:51.516743 4458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:51.519835 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "62de1332aebd80bb62d6ad5ed2a22c0f"
server_key_iv: "5431416284d40e0bf231fa76e3a41105"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.520619 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "62de1332aebd80bb62d6ad5ed2a22c0f"
server_key_iv: "5431416284d40e0bf231fa76e3a41105"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.525776 4458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:18:51.529294 4474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:51.530618 4458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:18:51.530797 4458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e"
format_stamp: "Formatted at 2026-08-09 06:18:51 on dist-test-slave-2w3w"
server_key: "62de1332aebd80bb62d6ad5ed2a22c0f"
server_key_iv: "5431416284d40e0bf231fa76e3a41105"
server_key_version: "encryptionkey@0"
I20260809 06:18:51.531049 4458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:51.547775 4458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:51.552112 4458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:51.552413 4458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:51.553400 4458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:51.554746 4458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:51.554821 4458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:51.554975 4458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:51.555029 4458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 06:18:51.591488 4458 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:38065
I20260809 06:18:51.591578 4594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:38065 every 8 connection(s)
I20260809 06:18:51.592818 4458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:18:51.602270 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4458
I20260809 06:18:51.602468 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:18:51.603014 3621 external_mini_cluster.cc:1428] Setting key 48f439188497aa9148fc8774f8880625
Aug 09 06:18:51 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256331, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:51.614042 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:51.595544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58145 (local address 127.3.137.126:34697)
0809 06:18:51.595731 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:51.595736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:51.596865 (+ 1129us) server_negotiation.cc:407] Connection header received
0809 06:18:51.598637 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:51.598642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:51.598734 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:51.598851 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:51.600327 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.601054 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.602712 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.603280 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.603435 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:51.607099 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:51.607108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:51.607133 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:51.607185 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:51.610877 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:51.611649 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:51.611657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:51.611663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:51.611802 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:51.612250 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:51.612256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:51.612259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:51.612557 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:51.612725 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:51.613614 (+ 889us) server_negotiation.cc:299] Negotiation successful
0809 06:18:51.613797 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:18:51.615628 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:51.596002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34697 (local address 127.3.137.66:58145)
0809 06:18:51.596698 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:51.596733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:51.598363 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:51.599105 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:51.599113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:51.599612 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:51.600147 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:51.600162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.601222 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.601228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:51.602562 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:51.602578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:51.603551 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:51.603560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:51.603791 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:51.604492 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:51.604527 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:51.606804 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:51.611065 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:51.611071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:51.611085 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:51.611512 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:51.611924 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:51.611929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:51.611933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:51.612128 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:51.612697 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:51.612705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:51.613181 (+ 476us) client_negotiation.cc:769] Sending connection context
0809 06:18:51.613497 (+ 316us) client_negotiation.cc:240] Negotiation successful
0809 06:18:51.614108 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
I20260809 06:18:51.617226 4595 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34697
I20260809 06:18:51.617559 4595 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:51.618343 4595 heartbeater.cc:511] Master 127.3.137.126:34697 requested a full tablet report, sending...
I20260809 06:18:51.620131 4247 ts_manager.cc:194] Registered new tserver with Master: 1188fe4c5fe443f3a10cd3a6c89a8f2e (127.3.137.66:38065)
I20260809 06:18:51.621030 4247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:58145
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:18:51.651263 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:51.975845 4602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:51.976352 4602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:51.976552 4602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:51.984617 4602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:51.984807 4602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:51.985001 4602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:18:51.995272 4602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:51.996618 4602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:51.998027 4602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:52.012802 4607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:52.012753 4608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:52.012730 4610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:52.013700 4602 server_base.cc:1034] running on GCE node
I20260809 06:18:52.014322 4602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:52.015411 4602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:52.016656 4602 hybrid_clock.cc:648] HybridClock initialized: now 1786256332016622 us; error 47 us; skew 500 ppm
I20260809 06:18:52.016978 4602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:52.020366 4602 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:18:52.021829 4602 webserver.cc:460] Webserver started at http://127.3.137.67:42801/ using document root <none> and password file <none>
I20260809 06:18:52.022547 4602 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:52.022701 4602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:52.023090 4602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:52.026132 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "424e98d47d9b4c23881f7f5b7ea51807"
format_stamp: "Formatted at 2026-08-09 06:18:52 on dist-test-slave-2w3w"
server_key: "e63b47f838a33062b2c77b42416186c8"
server_key_iv: "5d727bb55a058325cccfbeac1e840ddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:52.027036 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "424e98d47d9b4c23881f7f5b7ea51807"
format_stamp: "Formatted at 2026-08-09 06:18:52 on dist-test-slave-2w3w"
server_key: "e63b47f838a33062b2c77b42416186c8"
server_key_iv: "5d727bb55a058325cccfbeac1e840ddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:52.032619 4602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:18:52.037431 4619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:52.038914 4602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:18:52.039126 4602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "424e98d47d9b4c23881f7f5b7ea51807"
format_stamp: "Formatted at 2026-08-09 06:18:52 on dist-test-slave-2w3w"
server_key: "e63b47f838a33062b2c77b42416186c8"
server_key_iv: "5d727bb55a058325cccfbeac1e840ddb"
server_key_version: "encryptionkey@0"
I20260809 06:18:52.039405 4602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:52.055935 4602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:52.060519 4602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:52.060770 4602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:52.061695 4602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:18:52.062989 4602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:52.063063 4602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:52.063155 4602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:52.063212 4602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:52.097869 4602 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:38043
I20260809 06:18:52.097924 4736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:38043 every 8 connection(s)
I20260809 06:18:52.099294 4602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:18:52.100203 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4602
I20260809 06:18:52.100435 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:18:52.100941 3621 external_mini_cluster.cc:1428] Setting key cc116dd212891a4898ed51686b4bace2
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:52.122612 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.102316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:51341 (local address 127.3.137.126:34697)
0809 06:18:52.102549 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.102555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.103709 (+ 1154us) server_negotiation.cc:407] Connection header received
0809 06:18:52.105916 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.105921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.106004 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.106110 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.107891 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.108638 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.111476 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.111885 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.111988 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.116166 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.116170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.116174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.116207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.119384 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.120174 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.120185 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.120187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.120322 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.120793 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.120803 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.120805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.121090 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.121257 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.121999 (+ 742us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.122231 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:18:52.123695 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.102677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34697 (local address 127.3.137.67:51341)
0809 06:18:52.103559 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.103610 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.105622 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.106324 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.106332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.106850 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:52.107496 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.107515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.108801 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.108809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.111322 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.111342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.112112 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.112121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.112511 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.113326 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:52.113348 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:52.115922 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:52.119547 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:52.119556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:52.119575 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:52.120033 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:52.120445 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:52.120450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:52.120454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:52.120662 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:52.121230 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:52.121238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:52.121609 (+ 371us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.121888 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.122457 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":354,"threads_started":1}
I20260809 06:18:52.125162 4737 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34697
I20260809 06:18:52.125481 4737 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:52.126214 4737 heartbeater.cc:511] Master 127.3.137.126:34697 requested a full tablet report, sending...
I20260809 06:18:52.127821 4247 ts_manager.cc:194] Registered new tserver with Master: 424e98d47d9b4c23881f7f5b7ea51807 (127.3.137.67:38043)
I20260809 06:18:52.128913 4247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:51341
I20260809 06:18:52.130154 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:18:52.134697 4451 heartbeater.cc:503] Master 127.3.137.126:34697 was elected leader, sending a full tablet report...
I20260809 06:18:52.151561 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.137584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.3.137.126:34697)
0809 06:18:52.137742 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.137747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.138304 (+ 557us) server_negotiation.cc:407] Connection header received
0809 06:18:52.138425 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.138429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.138521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.138617 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.140291 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.141020 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.142267 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.142834 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.142988 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.145770 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.145783 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.145786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.145818 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.149056 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.149818 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.149822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.149824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.149951 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.150287 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.150290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.150292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.150625 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.150829 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.151153 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.151313 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:18:52.158463 4247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:18:52.165812 4247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:18:52.194204 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.178702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44005 (local address 127.0.0.1:59524)
0809 06:18:52.180633 (+ 1931us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.180672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.180862 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.182212 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.182223 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.182277 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.182791 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.182807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.185519 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.185528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.189067 (+ 3539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.189092 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.192588 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.192598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.192937 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.193051 (+ 114us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.193243 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.193545 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"mutex_wait_us":52,"thread_start_us":1545,"threads_started":1}
I20260809 06:18:52.194195 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.179134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.3.137.65:44005)
0809 06:18:52.181336 (+ 2202us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.181350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.181373 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:18:52.181450 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.181457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.181702 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.181895 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.183087 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.185303 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.189348 (+ 4045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.191845 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.192046 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.192364 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.193663 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.193846 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2051,"thread_start_us":699,"threads_started":1}
I20260809 06:18:52.195168 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.179493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38065 (local address 127.0.0.1:54126)
0809 06:18:52.181613 (+ 2120us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.181631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.181740 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.182805 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.182813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.182855 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.183464 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.183480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.186131 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.186140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.189330 (+ 3190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.189353 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.192515 (+ 3162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.192523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.194380 (+ 1857us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.194413 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.194521 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.194680 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"mutex_wait_us":1524,"thread_start_us":1848,"threads_started":1}
I20260809 06:18:52.195605 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.180122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.3.137.66:38065)
0809 06:18:52.181277 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.181286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.181874 (+ 588us) server_negotiation.cc:407] Connection header received
0809 06:18:52.182011 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.182021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.182284 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.182512 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.183718 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.185913 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.189614 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.191805 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.191995 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.192293 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.195029 (+ 2736us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.195237 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":820,"threads_started":1}
I20260809 06:18:52.199138 4382 tablet_service.cc:1467] Processing CreateTablet for tablet b404b031e2a5482dadd1d3fb0c7057dc (DEFAULT_TABLE table=test-table [id=9cbe8291951d4a2489bed7b09817652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:52.199273 4525 tablet_service.cc:1467] Processing CreateTablet for tablet b404b031e2a5482dadd1d3fb0c7057dc (DEFAULT_TABLE table=test-table [id=9cbe8291951d4a2489bed7b09817652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:52.200682 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404b031e2a5482dadd1d3fb0c7057dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:52.200690 4525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404b031e2a5482dadd1d3fb0c7057dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:52.203338 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.181692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38043 (local address 127.0.0.1:42144)
0809 06:18:52.193309 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.193328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.193460 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.195765 (+ 2305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.195770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.195811 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.196150 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.196164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.198202 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.198207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.200297 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.200311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.202513 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.202519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.202688 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.202726 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.202837 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.203001 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11435,"thread_start_us":11270,"threads_started":1}
I20260809 06:18:52.203449 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.182007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.3.137.67:38043)
0809 06:18:52.194999 (+ 12992us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.195009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.195047 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:18:52.195156 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.195166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.195482 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.195682 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.196389 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.198031 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.200497 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.202281 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.202444 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.202727 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.202910 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.203133 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12784,"thread_start_us":12611,"threads_started":1}
I20260809 06:18:52.207468 4670 tablet_service.cc:1467] Processing CreateTablet for tablet b404b031e2a5482dadd1d3fb0c7057dc (DEFAULT_TABLE table=test-table [id=9cbe8291951d4a2489bed7b09817652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:52.209010 4670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404b031e2a5482dadd1d3fb0c7057dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:52.216856 4756 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Bootstrap starting.
I20260809 06:18:52.219544 4757 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Bootstrap starting.
I20260809 06:18:52.221191 4756 tablet_bootstrap.cc:654] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:52.222308 4756 log.cc:824] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:52.223255 4757 tablet_bootstrap.cc:654] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:52.224556 4757 log.cc:824] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:52.226646 4761 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Bootstrap starting.
I20260809 06:18:52.227070 4756 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: No bootstrap required, opened a new log
I20260809 06:18:52.227415 4756 ts_tablet_manager.cc:1397] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260809 06:18:52.227634 4757 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: No bootstrap required, opened a new log
I20260809 06:18:52.227886 4757 ts_tablet_manager.cc:1397] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260809 06:18:52.230510 4761 tablet_bootstrap.cc:654] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:52.231803 4761 log.cc:824] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:52.235636 4761 tablet_bootstrap.cc:492] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: No bootstrap required, opened a new log
I20260809 06:18:52.235911 4761 ts_tablet_manager.cc:1397] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 06:18:52.236649 4757 raft_consensus.cc:348] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.237030 4757 raft_consensus.cc:374] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:52.237156 4757 raft_consensus.cc:729] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1188fe4c5fe443f3a10cd3a6c89a8f2e, State: Initialized, Role: FOLLOWER
I20260809 06:18:52.237777 4757 consensus_queue.cc:260] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.238063 4756 raft_consensus.cc:348] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.238549 4756 raft_consensus.cc:374] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:52.238699 4756 raft_consensus.cc:729] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a9279bcb8b4d7989f5525daaab3262, State: Initialized, Role: FOLLOWER
I20260809 06:18:52.238977 4595 heartbeater.cc:503] Master 127.3.137.126:34697 was elected leader, sending a full tablet report...
I20260809 06:18:52.239404 4756 consensus_queue.cc:260] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.239851 4757 ts_tablet_manager.cc:1428] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260809 06:18:52.241500 4756 ts_tablet_manager.cc:1428] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 06:18:52.246625 4761 raft_consensus.cc:348] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.247040 4761 raft_consensus.cc:374] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:52.247155 4761 raft_consensus.cc:729] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424e98d47d9b4c23881f7f5b7ea51807, State: Initialized, Role: FOLLOWER
I20260809 06:18:52.247776 4761 consensus_queue.cc:260] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.248773 4737 heartbeater.cc:503] Master 127.3.137.126:34697 was elected leader, sending a full tablet report...
I20260809 06:18:52.249370 4761 ts_tablet_manager.cc:1428] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260809 06:18:52.289440 4763 raft_consensus.cc:482] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:18:52.289773 4764 raft_consensus.cc:482] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:18:52.289791 4763 raft_consensus.cc:504] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.290045 4764 raft_consensus.cc:504] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.291879 4763 leader_election.cc:288] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a9279bcb8b4d7989f5525daaab3262 (127.3.137.65:44005), 424e98d47d9b4c23881f7f5b7ea51807 (127.3.137.67:38043)
I20260809 06:18:52.291929 4764 leader_election.cc:288] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1188fe4c5fe443f3a10cd3a6c89a8f2e (127.3.137.66:38065), 424e98d47d9b4c23881f7f5b7ea51807 (127.3.137.67:38043)
I20260809 06:18:52.305199 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.292922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:40307 (local address 127.3.137.65:44005)
0809 06:18:52.293170 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.293176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.294620 (+ 1444us) server_negotiation.cc:407] Connection header received
0809 06:18:52.294930 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.294938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.295054 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.295128 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.296167 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.297730 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.301912 (+ 4182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.303894 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.304072 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.304187 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.304807 (+ 620us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.304932 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:18:52.305281 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.292211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44005 (local address 127.3.137.66:40307)
0809 06:18:52.294500 (+ 2289us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.294523 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.294752 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.295369 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.295378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.295411 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.295965 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.295981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.297957 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.297964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.301648 (+ 3684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.301669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.304432 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.304438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.304610 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.304644 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.304770 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.304905 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"thread_start_us":1884,"threads_started":1}
I20260809 06:18:52.306543 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a9279bcb8b4d7989f5525daaab3262" is_pre_election: true
I20260809 06:18:52.307091 4403 raft_consensus.cc:2454] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in term 0.
I20260809 06:18:52.307873 4477 leader_election.cc:302] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1188fe4c5fe443f3a10cd3a6c89a8f2e, d3a9279bcb8b4d7989f5525daaab3262; no voters:
I20260809 06:18:52.308382 4763 raft_consensus.cc:2781] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:18:52.308540 4763 raft_consensus.cc:482] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:18:52.308655 4763 raft_consensus.cc:3037] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:52.308753 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.292318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38065 (local address 127.3.137.65:46001)
0809 06:18:52.293218 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.293236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.293361 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.299044 (+ 5683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.299052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.299094 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.299617 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.299632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.302401 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.302409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.305079 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.305091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.307778 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.307784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.307964 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.307999 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.308230 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.308371 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":543,"threads_started":1}
I20260809 06:18:52.309155 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.292487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:46001 (local address 127.3.137.66:38065)
0809 06:18:52.293437 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.293442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.293461 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:18:52.293527 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.293533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.293605 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.293702 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.300588 (+ 6886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.302149 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.306254 (+ 4105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.307364 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.307512 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.307587 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.308169 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.308383 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":80}
I20260809 06:18:52.310240 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "d3a9279bcb8b4d7989f5525daaab3262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" is_pre_election: true
I20260809 06:18:52.313174 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.293442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:40371 (local address 127.3.137.67:38043)
0809 06:18:52.293930 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.293939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.305628 (+ 11689us) server_negotiation.cc:407] Connection header received
0809 06:18:52.305845 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.305852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.305937 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.306043 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.306606 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.307667 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.310433 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.311629 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.311843 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.311915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.312794 (+ 879us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.312925 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260809 06:18:52.313174 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.293230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38043 (local address 127.3.137.66:40371)
0809 06:18:52.305545 (+ 12315us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.305562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.305724 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.306202 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.306209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.306236 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.306441 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.306452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.307858 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.307866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.310239 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.310255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.311842 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.311848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.312617 (+ 769us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.312642 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.312762 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.312897 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12175,"thread_start_us":12871,"threads_started":1}
I20260809 06:18:52.313791 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.293557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:55893 (local address 127.3.137.67:38043)
0809 06:18:52.304165 (+ 10608us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.304171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.304955 (+ 784us) server_negotiation.cc:407] Connection header received
0809 06:18:52.305037 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.305043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.305120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.305227 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:52.307671 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.308947 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.311311 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.312684 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.312890 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.312955 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.313250 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.313410 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10453,"thread_start_us":10608,"threads_started":1}
I20260809 06:18:52.314397 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.293294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38043 (local address 127.3.137.65:55893)
0809 06:18:52.304713 (+ 11419us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:52.304732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:52.304849 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:52.305602 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:52.305614 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:52.305651 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:52.306091 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.307462 (+ 1371us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.309113 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.309118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.311130 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:52.311145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.312906 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.312913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:52.313081 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.313104 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:18:52.313212 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:18:52.314102 (+ 890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11290,"thread_start_us":11114,"threads_started":1}
I20260809 06:18:52.314399 4691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424e98d47d9b4c23881f7f5b7ea51807" is_pre_election: true
I20260809 06:18:52.313778 4763 raft_consensus.cc:504] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.315111 4546 raft_consensus.cc:2379] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a9279bcb8b4d7989f5525daaab3262 in current term 1: Already voted for candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in this term.
I20260809 06:18:52.315058 4691 raft_consensus.cc:2454] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in term 0.
I20260809 06:18:52.315114 4690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "d3a9279bcb8b4d7989f5525daaab3262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424e98d47d9b4c23881f7f5b7ea51807" is_pre_election: true
I20260809 06:18:52.315412 4690 raft_consensus.cc:2454] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a9279bcb8b4d7989f5525daaab3262 in term 0.
I20260809 06:18:52.316064 4763 leader_election.cc:288] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [CANDIDATE]: Term 1 election: Requested vote from peers d3a9279bcb8b4d7989f5525daaab3262 (127.3.137.65:44005), 424e98d47d9b4c23881f7f5b7ea51807 (127.3.137.67:38043)
I20260809 06:18:52.316497 4333 leader_election.cc:302] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424e98d47d9b4c23881f7f5b7ea51807, d3a9279bcb8b4d7989f5525daaab3262; no voters:
I20260809 06:18:52.316562 4690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424e98d47d9b4c23881f7f5b7ea51807"
I20260809 06:18:52.316757 4690 raft_consensus.cc:3037] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:52.317016 4764 raft_consensus.cc:2781] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:18:52.317174 4764 raft_consensus.cc:482] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:18:52.317155 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a9279bcb8b4d7989f5525daaab3262"
I20260809 06:18:52.317322 4764 raft_consensus.cc:3037] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 0 FOLLOWER]: Advancing to term 1
W20260809 06:18:52.344939 4597 tablet.cc:2369] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:18:52.351085 4738 tablet.cc:2369] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:18:52.356351 4452 tablet.cc:2369] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:18:52.413707 4690 raft_consensus.cc:2454] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in term 1.
I20260809 06:18:52.413734 4764 raft_consensus.cc:504] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.414387 4403 raft_consensus.cc:2379] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in current term 1: Already voted for candidate d3a9279bcb8b4d7989f5525daaab3262 in this term.
I20260809 06:18:52.414494 4478 leader_election.cc:302] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1188fe4c5fe443f3a10cd3a6c89a8f2e, 424e98d47d9b4c23881f7f5b7ea51807; no voters:
I20260809 06:18:52.414666 4764 leader_election.cc:288] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [CANDIDATE]: Term 1 election: Requested vote from peers 1188fe4c5fe443f3a10cd3a6c89a8f2e (127.3.137.66:38065), 424e98d47d9b4c23881f7f5b7ea51807 (127.3.137.67:38043)
I20260809 06:18:52.414921 4763 raft_consensus.cc:2781] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:52.415529 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "d3a9279bcb8b4d7989f5525daaab3262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e"
I20260809 06:18:52.416626 4690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404b031e2a5482dadd1d3fb0c7057dc" candidate_uuid: "d3a9279bcb8b4d7989f5525daaab3262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424e98d47d9b4c23881f7f5b7ea51807"
I20260809 06:18:52.416834 4763 raft_consensus.cc:686] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [term 1 LEADER]: Becoming Leader. State: Replica: 1188fe4c5fe443f3a10cd3a6c89a8f2e, State: Running, Role: LEADER
I20260809 06:18:52.417047 4690 raft_consensus.cc:2379] T b404b031e2a5482dadd1d3fb0c7057dc P 424e98d47d9b4c23881f7f5b7ea51807 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a9279bcb8b4d7989f5525daaab3262 in current term 1: Already voted for candidate 1188fe4c5fe443f3a10cd3a6c89a8f2e in this term.
I20260809 06:18:52.417788 4763 consensus_queue.cc:237] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } }
I20260809 06:18:52.420634 4333 leader_election.cc:302] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a9279bcb8b4d7989f5525daaab3262; no voters: 1188fe4c5fe443f3a10cd3a6c89a8f2e, 424e98d47d9b4c23881f7f5b7ea51807
I20260809 06:18:52.420970 4764 raft_consensus.cc:2726] T b404b031e2a5482dadd1d3fb0c7057dc P d3a9279bcb8b4d7989f5525daaab3262 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 06:18:52.422513 4246 catalog_manager.cc:5495] T b404b031e2a5482dadd1d3fb0c7057dc P 1188fe4c5fe443f3a10cd3a6c89a8f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1188fe4c5fe443f3a10cd3a6c89a8f2e (127.3.137.66). New cstate: current_term: 1 leader_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9279bcb8b4d7989f5525daaab3262" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1188fe4c5fe443f3a10cd3a6c89a8f2e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "424e98d47d9b4c23881f7f5b7ea51807" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38043 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:52.450902 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.435308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.3.137.65:44005)
0809 06:18:52.435489 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.435493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.436071 (+ 578us) server_negotiation.cc:407] Connection header received
0809 06:18:52.436277 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.436283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.436362 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.436483 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.438002 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.438883 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.440211 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.440650 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.440792 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.445072 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.445090 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.445105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.445146 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.448003 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.448695 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.448704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.448706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.448841 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.449224 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.449230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.449233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.449619 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.449992 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.450299 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.450546 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:52.536224 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.520856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.3.137.65:44005)
0809 06:18:52.521146 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.521155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.521816 (+ 661us) server_negotiation.cc:407] Connection header received
0809 06:18:52.521986 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.521991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.522066 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.522162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.523802 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.524677 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.526031 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.526466 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.526569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.530372 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.530377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.530381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.530416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.533491 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.534126 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.534131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.534134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.534327 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.534706 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.534719 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.534722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.535180 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.535423 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.535663 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.535886 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:52.639675 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.623900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.3.137.65:44005)
0809 06:18:52.624165 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.624174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.624594 (+ 420us) server_negotiation.cc:407] Connection header received
0809 06:18:52.624772 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.624780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.624878 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.625013 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.626572 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.627394 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.628712 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.629148 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.629244 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.633659 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.633664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.633668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.633706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.636836 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.637511 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.637515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.637518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.637640 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.637959 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.637962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.637964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.638544 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.638786 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.639155 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.639402 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 06:18:52.641919 4358 tablet_service.cc:2908] Rejecting scan request for tablet b404b031e2a5482dadd1d3fb0c7057dc: Uninitialized: clean time has not yet been initialized
I20260809 06:18:52.646173 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4312
I20260809 06:18:52.659596 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.647021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.3.137.65:44005)
0809 06:18:52.647194 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.647198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.647697 (+ 499us) server_negotiation.cc:407] Connection header received
0809 06:18:52.647850 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.647853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.647921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.647998 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.649267 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.649938 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.651358 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.651766 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.651911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.654539 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.654544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.654547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.654576 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.657149 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.657847 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.657850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.657853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.657968 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.658290 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.658293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.658295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.658546 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.658842 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.659185 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.659349 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:18:52.660295 4446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:52.760892 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4312
I20260809 06:18:52.781549 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4458
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:18:52.796718 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.782327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.3.137.66:38065)
0809 06:18:52.782535 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.782542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.782646 (+ 104us) server_negotiation.cc:407] Connection header received
0809 06:18:52.782862 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.782866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.782931 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.783095 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.784302 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.785018 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.786338 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.786739 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.786888 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.790816 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.790830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.790847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.790886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.794003 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.794767 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.794812 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.794815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.795014 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.795381 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.795384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.795386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.795765 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.796010 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.796272 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.796460 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:18:52.797454 4590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:52.892258 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4458
I20260809 06:18:52.911602 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4602
Aug 09 06:18:52 dist-test-slave-2w3w krb5kdc[4197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:18:52.926759 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:52.912408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.3.137.67:38043)
0809 06:18:52.912974 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:52.912979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:52.912996 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:18:52.913052 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:52.913056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:52.913114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:52.913205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:52.914454 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.915289 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.916573 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:52.916957 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:52.917052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:52.921263 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:52.921278 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:52.921294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:52.921335 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:52.924233 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.924833 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.924840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.924842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.924996 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:52.925400 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:52.925404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:52.925406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:52.925851 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:52.926137 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:52.926333 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 06:18:52.926502 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":322,"threads_started":1}
I20260809 06:18:52.927464 4732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:53.020416 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4602
I20260809 06:18:53.040470 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4212
I20260809 06:18:53.041836 4278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:53.159341 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4212
2026-08-09T06:18:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4384 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:18:53 dist-test-slave-2w3w krb5kdc[4795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:18:53 dist-test-slave-2w3w krb5kdc[4795](info): set up 2 sockets
Aug 09 06:18:53 dist-test-slave-2w3w krb5kdc[4795](info): commencing operation
krb5kdc: starting...
W20260809 06:18:54.952607 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.747s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:18:55 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:18:55.155032 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:34465
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:55.475926 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:55.476222 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:55.476378 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:55.483752 4808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:18:55.483858 4808 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:18:55.483929 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:55.484019 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:55.484086 4808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:18:55.484171 4808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:18:55.494184 4808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:34465
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:55.495584 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:55.496901 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:55.508946 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:55.509083 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:55.509733 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:55.509892 4808 server_base.cc:1034] running on GCE node
I20260809 06:18:55.510735 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:55.512307 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:55.513563 4808 hybrid_clock.cc:648] HybridClock initialized: now 1786256335513497 us; error 44 us; skew 500 ppm
I20260809 06:18:55.513907 4808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:55 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:55.520327 4808 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:18:55.521880 4808 webserver.cc:460] Webserver started at http://127.3.137.126:46857/ using document root <none> and password file <none>
I20260809 06:18:55.522626 4808 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:55.522714 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:55.523041 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:55.525920 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "80368557fe8c4d50a532b71ab9f7d3a3"
format_stamp: "Formatted at 2026-08-09 06:18:55 on dist-test-slave-2w3w"
server_key: "a3a4ebd150b04ab391d1970bddc76a7c"
server_key_iv: "5983329356449e5cdf73516153cee5c0"
server_key_version: "encryptionkey@0"
I20260809 06:18:55.526692 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "80368557fe8c4d50a532b71ab9f7d3a3"
format_stamp: "Formatted at 2026-08-09 06:18:55 on dist-test-slave-2w3w"
server_key: "a3a4ebd150b04ab391d1970bddc76a7c"
server_key_iv: "5983329356449e5cdf73516153cee5c0"
server_key_version: "encryptionkey@0"
I20260809 06:18:55.531840 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:18:55.535410 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:55.536757 4808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:18:55.536940 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "80368557fe8c4d50a532b71ab9f7d3a3"
format_stamp: "Formatted at 2026-08-09 06:18:55 on dist-test-slave-2w3w"
server_key: "a3a4ebd150b04ab391d1970bddc76a7c"
server_key_iv: "5983329356449e5cdf73516153cee5c0"
server_key_version: "encryptionkey@0"
I20260809 06:18:55.537169 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:55.559093 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:55.563331 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:55.563632 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:55.587173 4808 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:34465
I20260809 06:18:55.587208 4889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:34465 every 8 connection(s)
I20260809 06:18:55.588590 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:18:55.592577 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:55.593958 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4808
I20260809 06:18:55.594213 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:18:55.595124 3621 external_mini_cluster.cc:1428] Setting key 898ec1fb7a9a6099bbfbbd21f7ed4056
I20260809 06:18:55.607604 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Bootstrap starting.
Aug 09 06:18:55 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:55.611394 4890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:55.612506 4890 log.cc:824] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:55.616091 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: No bootstrap required, opened a new log
I20260809 06:18:55.619685 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:55.597262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.3.137.126:34465)
0809 06:18:55.598097 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:55.598111 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:55.598153 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:18:55.599901 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:55.599929 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:55.600328 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:55.600656 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:55.602422 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:55.603794 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:55.605583 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:55.606242 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:55.606388 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:55.611185 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:55.611193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:55.611207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:55.611242 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:55.615197 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:55.615864 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:55.615872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:55.615880 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:55.616110 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:55.616547 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:55.616554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:55.616558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:55.617235 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:55.617549 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:55.617823 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 06:18:55.618282 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":403,"threads_started":1}
I20260809 06:18:55.625505 4890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } }
I20260809 06:18:55.625810 4890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:55.625895 4890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80368557fe8c4d50a532b71ab9f7d3a3, State: Initialized, Role: FOLLOWER
I20260809 06:18:55.626354 4890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } }
I20260809 06:18:55.626554 4890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:18:55.626649 4890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:18:55.626770 4890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:55.629133 4890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } }
I20260809 06:18:55.629634 4890 leader_election.cc:302] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80368557fe8c4d50a532b71ab9f7d3a3; no voters:
I20260809 06:18:55.630209 4890 leader_election.cc:288] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:18:55.630369 4895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:55.631108 4895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 80368557fe8c4d50a532b71ab9f7d3a3, State: Running, Role: LEADER
I20260809 06:18:55.631660 4895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } }
I20260809 06:18:55.632314 4890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:18:55.635629 4896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } } }
I20260809 06:18:55.635713 4897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80368557fe8c4d50a532b71ab9f7d3a3. Latest consensus state: current_term: 1 leader_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80368557fe8c4d50a532b71ab9f7d3a3" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 34465 } } }
I20260809 06:18:55.636025 4896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:55.636114 4897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3 [sys.catalog]: This master's current role is: LEADER
I20260809 06:18:55.638188 4902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:18:55.643206 4902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:18:55.651335 4902 catalog_manager.cc:1347] Generated new cluster ID: 5e001f4c70354d71a1d6fd6fa475a023
I20260809 06:18:55.651450 4902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:18:55.664983 4902 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:18:55.666815 4902 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:18:55.677722 4902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Generated new TSK 0
I20260809 06:18:55.678810 4902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:18:55.731757 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34465
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:56.064487 4918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:56.064778 4918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:56.064949 4918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:56.072420 4918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:56.072525 4918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:56.072722 4918 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:18:56.072811 4918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:18:56.082561 4918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:34465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:56.083818 4918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:56.085150 4918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:56.098532 4923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:56.098500 4924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:56.098748 4926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:56.099507 4918 server_base.cc:1034] running on GCE node
I20260809 06:18:56.100210 4918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:56.101310 4918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:56.102594 4918 hybrid_clock.cc:648] HybridClock initialized: now 1786256336102504 us; error 32 us; skew 500 ppm
I20260809 06:18:56.102882 4918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:56 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:56.106199 4918 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:18:56.107728 4918 webserver.cc:460] Webserver started at http://127.3.137.65:46837/ using document root <none> and password file <none>
I20260809 06:18:56.108469 4918 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:56.108620 4918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:56.108978 4918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:56.112200 4918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "7aae5c624cca4755b82d7dba5ee44b57"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "5560d9ebcc157f0eba4be02c607ed86c"
server_key_iv: "c06a8cd010c3d89b46ddbf045b3c24a4"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.113113 4918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "7aae5c624cca4755b82d7dba5ee44b57"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "5560d9ebcc157f0eba4be02c607ed86c"
server_key_iv: "c06a8cd010c3d89b46ddbf045b3c24a4"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.118464 4918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:18:56.122139 4933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:56.123574 4918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:18:56.123764 4918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "7aae5c624cca4755b82d7dba5ee44b57"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "5560d9ebcc157f0eba4be02c607ed86c"
server_key_iv: "c06a8cd010c3d89b46ddbf045b3c24a4"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.124004 4918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:56.136742 4918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:56.141131 4918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:56.141412 4918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:56.144192 4918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:56.144305 4918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:56.144471 4918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:56.144562 4918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:18:56 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:56.173465 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.145591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.65:47237)
0809 06:18:56.146510 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:56.146565 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:56.148953 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:56.149915 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:56.149931 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:56.150853 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:56.151916 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.151952 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.153751 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.153756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.156734 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.156755 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.157863 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.157874 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.158231 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.159144 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:56.159177 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:56.163255 (+ 4078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:56.167679 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.167690 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.167713 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.168454 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.169033 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.169040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.169044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.169329 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.170105 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:56.170113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:56.170498 (+ 385us) client_negotiation.cc:769] Sending connection context
0809 06:18:56.170928 (+ 430us) client_negotiation.cc:240] Negotiation successful
0809 06:18:56.171641 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":418,"threads_started":1}
I20260809 06:18:56.175243 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.145053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:47237 (local address 127.3.137.126:34465)
0809 06:18:56.147336 (+ 2283us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:56.147343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:56.147369 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:18:56.149357 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:56.149365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:56.149471 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:56.149604 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:56.152152 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.153523 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.156925 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.157551 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.157703 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.163578 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:56.163584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:56.163587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:56.163621 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:56.167458 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.168642 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.168646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.168649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.168861 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.169479 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.169485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.169491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.169936 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:56.170118 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:56.174512 (+ 4394us) server_negotiation.cc:299] Negotiation successful
0809 06:18:56.174818 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2141,"thread_start_us":2029,"threads_started":1}
I20260809 06:18:56.189193 4918 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:42583
I20260809 06:18:56.189209 5065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:42583 every 8 connection(s)
I20260809 06:18:56.190505 4918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:18:56.193164 5066 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34465
I20260809 06:18:56.193416 5066 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:56.194072 5066 heartbeater.cc:511] Master 127.3.137.126:34465 requested a full tablet report, sending...
I20260809 06:18:56.194523 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 4918
I20260809 06:18:56.194811 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:18:56.196180 3621 external_mini_cluster.cc:1428] Setting key 7f4af3c1e63f55249061ca064a54f246
I20260809 06:18:56.196233 4840 ts_manager.cc:194] Registered new tserver with Master: 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583)
I20260809 06:18:56.198105 4840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:47237
I20260809 06:18:56.200281 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.183761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:52369 (local address 127.3.137.126:34465)
0809 06:18:56.184008 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:56.184017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:56.184164 (+ 147us) server_negotiation.cc:407] Connection header received
0809 06:18:56.184423 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:56.184430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:56.184520 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:56.184641 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:56.186591 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.187588 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.189218 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.189750 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.189914 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.192935 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:56.192942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:56.192948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:56.193018 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:56.197034 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.197955 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.197960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.197962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.198163 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.198638 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.198643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.198645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.199034 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:56.199182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:56.199727 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 06:18:56.199962 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:18:56.201232 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.183385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.65:52369)
0809 06:18:56.184048 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:56.184070 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:56.184233 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:56.184850 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:56.184857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:56.185660 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:56.186393 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.186413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.187762 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.187768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.189044 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.189061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.189967 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.189974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.190172 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.190855 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:56.190874 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:56.192732 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:56.197217 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.197222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.197227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.197828 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.198327 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.198333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.198336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.198507 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.199151 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:56.199159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:56.199401 (+ 242us) client_negotiation.cc:769] Sending connection context
0809 06:18:56.199586 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 06:18:56.200035 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":555,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:18:56.242445 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34465
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:56.559769 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:56.560029 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:56.560127 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:56.567200 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:56.567307 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:56.567420 5070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:18:56.567510 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:18:56.577517 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:34465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:56.579068 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:56.580477 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:56.593411 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:56.593463 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:56.593928 5070 server_base.cc:1034] running on GCE node
W20260809 06:18:56.593621 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:56.594792 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:56.595860 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:56.597118 5070 hybrid_clock.cc:648] HybridClock initialized: now 1786256336597047 us; error 40 us; skew 500 ppm
I20260809 06:18:56.597458 5070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:56 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:56.601202 5070 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:18:56.604046 5070 webserver.cc:460] Webserver started at http://127.3.137.66:35307/ using document root <none> and password file <none>
I20260809 06:18:56.604759 5070 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:56.604879 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:56.605271 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:56.608358 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "0b8bd2bc9dc243f295a3b5f797e072cd"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "9c234c699a1354babdd2281b5804f657"
server_key_iv: "0da397cf8d727f1c29282d8892f2d5cc"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.609174 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "0b8bd2bc9dc243f295a3b5f797e072cd"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "9c234c699a1354babdd2281b5804f657"
server_key_iv: "0da397cf8d727f1c29282d8892f2d5cc"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.614028 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 06:18:56.617364 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:56.618608 5070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:18:56.618795 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "0b8bd2bc9dc243f295a3b5f797e072cd"
format_stamp: "Formatted at 2026-08-09 06:18:56 on dist-test-slave-2w3w"
server_key: "9c234c699a1354babdd2281b5804f657"
server_key_iv: "0da397cf8d727f1c29282d8892f2d5cc"
server_key_version: "encryptionkey@0"
I20260809 06:18:56.619060 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:56.655380 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:56.659441 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:56.659739 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:56.662380 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:56.662524 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:56.662706 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:56.662823 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:18:56 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:56.685904 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.663371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:33859 (local address 127.3.137.126:34465)
0809 06:18:56.663564 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:56.663570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:56.664980 (+ 1410us) server_negotiation.cc:407] Connection header received
0809 06:18:56.667483 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:56.667493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:56.667608 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:56.667740 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:56.669912 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.671286 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.673236 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.673672 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.673828 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.678416 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:56.678426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:56.678431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:56.678493 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:56.681892 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.683053 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.683061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.683066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.683224 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.683859 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.683866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.683871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.684232 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:56.684419 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:56.685487 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0809 06:18:56.685653 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:18:56.688235 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.663876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.66:33859)
0809 06:18:56.664777 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:56.664823 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:56.667075 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:56.668060 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:56.668071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:56.668861 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:56.669668 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.669719 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.671475 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.671484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.673048 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.673064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.673940 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.673947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.674209 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.675073 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:56.675095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:56.678165 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:56.682056 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.682069 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.682090 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.682866 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.683358 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.683368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.683372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.683696 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.684416 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:56.684427 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:56.685002 (+ 575us) client_negotiation.cc:769] Sending connection context
0809 06:18:56.685379 (+ 377us) client_negotiation.cc:240] Negotiation successful
0809 06:18:56.686068 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":419,"threads_started":1}
I20260809 06:18:56.710376 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.694995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.66:48989)
0809 06:18:56.695743 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:56.695776 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:56.695954 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:56.696375 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:56.696378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:56.696848 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:56.697145 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.697154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.699061 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.699065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.700126 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:56.700138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.701409 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.701414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:56.701535 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.702367 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:56.702395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:56.703736 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:56.707174 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.707179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.707183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.707854 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.708342 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:56.708350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:56.708380 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:56.708591 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:56.709481 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:56.709486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:56.709646 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 06:18:56.709755 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 06:18:56.710024 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":365,"threads_started":1}
I20260809 06:18:56.710563 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:56.694917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:48989 (local address 127.3.137.126:34465)
0809 06:18:56.695168 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:56.695176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:56.695898 (+ 722us) server_negotiation.cc:407] Connection header received
0809 06:18:56.696099 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:56.696107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:56.696216 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:56.696341 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:56.697289 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.698813 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.700275 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:56.701171 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:56.701356 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:56.703957 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:56.703962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:56.703966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:56.704007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:56.707035 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.708008 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.708014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.708019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.708193 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:56.708793 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:56.708798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:56.708801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:56.709342 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:56.709500 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:56.709936 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 06:18:56.710205 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260809 06:18:56.720232 5070 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:36797
I20260809 06:18:56.720252 5212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:36797 every 8 connection(s)
I20260809 06:18:56.721524 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:18:56.723073 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5070
I20260809 06:18:56.723315 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:18:56.723714 5213 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34465
I20260809 06:18:56.723930 3621 external_mini_cluster.cc:1428] Setting key b6096643b0397e9097f80231722edc7d
I20260809 06:18:56.723966 5213 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:56.724592 5213 heartbeater.cc:511] Master 127.3.137.126:34465 requested a full tablet report, sending...
I20260809 06:18:56.726186 4840 ts_manager.cc:194] Registered new tserver with Master: 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797)
I20260809 06:18:56.727480 4840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:33859
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:18:56.770658 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:34465
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:18:57.094421 5217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:18:57.094678 5217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:18:57.094856 5217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:18:57.102108 5217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:18:57.102205 5217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:18:57.102388 5217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:18:57.102520 5217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:18:57.112171 5217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:34465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:18:57.113432 5217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:18:57.114737 5217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:18:57.128212 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:57.128154 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:18:57.128106 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:18:57.129204 5217 server_base.cc:1034] running on GCE node
I20260809 06:18:57.129783 5217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:18:57.130671 5217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:18:57.131887 5217 hybrid_clock.cc:648] HybridClock initialized: now 1786256337131830 us; error 28 us; skew 500 ppm
I20260809 06:18:57.132171 5217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:18:57 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:18:57.135367 5217 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:18:57.136890 5217 webserver.cc:460] Webserver started at http://127.3.137.67:40921/ using document root <none> and password file <none>
I20260809 06:18:57.137624 5217 fs_manager.cc:359] Metadata directory not provided
I20260809 06:18:57.137714 5217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:18:57.137970 5217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:18:57.140954 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "da4859abb7c7420e8d6ee4a67c8149f4"
format_stamp: "Formatted at 2026-08-09 06:18:57 on dist-test-slave-2w3w"
server_key: "9fa0194ac54970d5ff657f5afad9577f"
server_key_iv: "7c9bbe587739e36c26d57a531bf495b6"
server_key_version: "encryptionkey@0"
I20260809 06:18:57.141803 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "da4859abb7c7420e8d6ee4a67c8149f4"
format_stamp: "Formatted at 2026-08-09 06:18:57 on dist-test-slave-2w3w"
server_key: "9fa0194ac54970d5ff657f5afad9577f"
server_key_iv: "7c9bbe587739e36c26d57a531bf495b6"
server_key_version: "encryptionkey@0"
I20260809 06:18:57.146781 5217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:18:57.150063 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:57.151302 5217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 06:18:57.151477 5217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "da4859abb7c7420e8d6ee4a67c8149f4"
format_stamp: "Formatted at 2026-08-09 06:18:57 on dist-test-slave-2w3w"
server_key: "9fa0194ac54970d5ff657f5afad9577f"
server_key_iv: "7c9bbe587739e36c26d57a531bf495b6"
server_key_version: "encryptionkey@0"
I20260809 06:18:57.151698 5217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:18:57.177467 5217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:18:57.181699 5217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:18:57.181978 5217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:18:57.184764 5217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:18:57.184888 5217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:18:57.185056 5217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:18:57.185118 5217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:18:57 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:57.202378 5066 heartbeater.cc:503] Master 127.3.137.126:34465 was elected leader, sending a full tablet report...
I20260809 06:18:57.210026 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.185859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.67:56121)
0809 06:18:57.186674 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.186708 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.189107 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.190025 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.190041 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.191016 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:57.192029 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.192061 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.193771 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.193800 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.195333 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.195350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.196633 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.196641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.196846 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.197745 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:57.197781 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:57.200573 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:57.205360 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:57.205369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:57.205383 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:57.205819 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:57.206242 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:57.206247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:57.206251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:57.206443 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:57.207021 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:57.207028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:57.207473 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.207898 (+ 425us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.208446 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":361,"threads_started":1}
I20260809 06:18:57.211077 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.185240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:56121 (local address 127.3.137.126:34465)
0809 06:18:57.185468 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.185476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.186842 (+ 1366us) server_negotiation.cc:407] Connection header received
0809 06:18:57.189537 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.189545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.189643 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.189800 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:57.192304 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.193584 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.195571 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.196325 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.196470 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.200842 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:57.200853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:57.200862 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:57.200919 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:57.205211 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.205982 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.205988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.205991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.206121 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.206570 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.206575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.206579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.206849 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:57.207032 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.210595 (+ 3563us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.210760 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:18:57.232573 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.217241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.3.137.67:53297)
0809 06:18:57.218003 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.218028 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.218182 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.218622 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.218625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.219304 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:57.219815 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.219834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.221459 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.221467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.223257 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.223273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.224144 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.224150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.224330 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.224996 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:57.225013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:57.225694 (+ 681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:57.229377 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:57.229383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:57.229389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:57.230016 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:57.230511 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:57.230520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:57.230524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:57.230723 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:57.231499 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:57.231508 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:57.231775 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.231978 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.232245 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":385,"threads_started":1}
I20260809 06:18:57.233403 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.217278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:53297 (local address 127.3.137.126:34465)
0809 06:18:57.217464 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.217471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.218146 (+ 675us) server_negotiation.cc:407] Connection header received
0809 06:18:57.218316 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.218325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.218425 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.218557 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:57.220021 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.221271 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.223423 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.223935 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.224104 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.225901 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:57.225907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:57.225911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:57.225950 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:57.229201 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.230175 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.230179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.230181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.230347 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.230894 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.230897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.230899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.231351 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:57.231528 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.232923 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.233155 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:18:57.243762 5217 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33195
I20260809 06:18:57.243845 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33195 every 8 connection(s)
I20260809 06:18:57.245030 5217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:18:57.247349 5358 heartbeater.cc:347] Connected to a master server at 127.3.137.126:34465
I20260809 06:18:57.247539 5358 heartbeater.cc:464] Registering TS with master...
I20260809 06:18:57.248113 5358 heartbeater.cc:511] Master 127.3.137.126:34465 requested a full tablet report, sending...
I20260809 06:18:57.249158 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5217
I20260809 06:18:57.249362 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:18:57.249596 4840 ts_manager.cc:194] Registered new tserver with Master: da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195)
I20260809 06:18:57.249830 3621 external_mini_cluster.cc:1428] Setting key b58a3360ef635affd54f5570d0f37d55
I20260809 06:18:57.250499 4840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:56121
I20260809 06:18:57.252931 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:18:57 dist-test-slave-2w3w krb5kdc[4795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:18:57 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:57.299832 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.286007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.3.137.126:34465)
0809 06:18:57.286217 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.286224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.286563 (+ 339us) server_negotiation.cc:407] Connection header received
0809 06:18:57.286702 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.286707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.286778 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.286869 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:57.288261 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.289037 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.290256 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.290757 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.290890 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.294665 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:57.294671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:57.294674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:57.294707 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:57.297120 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.297750 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.297759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.297762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.297885 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:57.298300 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:57.298306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:57.298309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:57.298585 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:57.298871 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.299216 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.299484 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 06:18:57.303815 4839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:18:57.311167 4839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:18:57.347956 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.323568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.3.137.65:42583)
0809 06:18:57.339226 (+ 15658us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.339234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.339258 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:18:57.339346 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.339352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.339556 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.340930 (+ 1374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.341144 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.342696 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.345261 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.346537 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.346672 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.346860 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.347474 (+ 614us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.347653 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15484,"thread_start_us":15306,"threads_started":1}
I20260809 06:18:57.348042 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.323351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:42583 (local address 127.0.0.1:52788)
0809 06:18:57.332366 (+ 9015us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.332426 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.332655 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.339764 (+ 7109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.339768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.339836 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.340355 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.340372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.342834 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.342840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.345115 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.345125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.347056 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.347067 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.347206 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.347295 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.347429 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.347653 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8704,"mutex_wait_us":97,"thread_start_us":13941,"threads_started":1}
I20260809 06:18:57.348031 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.323906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.3.137.67:33195)
0809 06:18:57.338663 (+ 14757us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.338673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.338706 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:18:57.339138 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.339146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.339373 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.339567 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.341116 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.342929 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.344868 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.346325 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.346475 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.346697 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.347594 (+ 897us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.347735 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14576,"thread_start_us":14380,"threads_started":1}
I20260809 06:18:57.348024 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.327235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.3.137.66:36797)
0809 06:18:57.338071 (+ 10836us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.338079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.338110 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:18:57.338921 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.338929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.339211 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.339421 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.340173 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.341499 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.344945 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.346323 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.346468 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.346671 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.347547 (+ 876us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.347724 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10561,"thread_start_us":10398,"threads_started":1}
I20260809 06:18:57.348973 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.326581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:36797 (local address 127.0.0.1:46780)
0809 06:18:57.337553 (+ 10972us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.337574 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.338725 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.339434 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.339441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.339469 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.339898 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.339912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.342129 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.342135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.344766 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.344782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.346846 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.346855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.347135 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.347281 (+ 146us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.347409 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.348502 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10854,"mutex_wait_us":992,"thread_start_us":5317,"threads_started":1}
I20260809 06:18:57.349045 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.323667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33195 (local address 127.0.0.1:33598)
0809 06:18:57.335487 (+ 11820us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.335505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.335610 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.339905 (+ 4295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.339912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.339983 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.340794 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.340807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.343097 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.343103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.344701 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.344714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.346898 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.346905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.347132 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.347298 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.347409 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.348595 (+ 1186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11672,"mutex_wait_us":1087,"thread_start_us":11516,"threads_started":1}
I20260809 06:18:57.351877 4989 tablet_service.cc:1467] Processing CreateTablet for tablet d10cdc849a2249f585673c30613e19be (DEFAULT_TABLE table=test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.352167 5140 tablet_service.cc:1467] Processing CreateTablet for tablet d10cdc849a2249f585673c30613e19be (DEFAULT_TABLE table=test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.353003 4989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10cdc849a2249f585673c30613e19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.353474 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10cdc849a2249f585673c30613e19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.354341 5286 tablet_service.cc:1467] Processing CreateTablet for tablet d10cdc849a2249f585673c30613e19be (DEFAULT_TABLE table=test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.355712 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10cdc849a2249f585673c30613e19be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.365327 5378 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Bootstrap starting.
I20260809 06:18:57.367769 5379 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Bootstrap starting.
I20260809 06:18:57.368956 5378 tablet_bootstrap.cc:654] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.369611 5380 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Bootstrap starting.
I20260809 06:18:57.370404 5378 log.cc:824] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:57.371148 5379 tablet_bootstrap.cc:654] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.372251 5379 log.cc:824] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:57.372857 5380 tablet_bootstrap.cc:654] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.373857 5380 log.cc:824] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:18:57.374598 5378 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: No bootstrap required, opened a new log
I20260809 06:18:57.374884 5378 ts_tablet_manager.cc:1397] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260809 06:18:57.376541 5379 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: No bootstrap required, opened a new log
I20260809 06:18:57.376808 5379 ts_tablet_manager.cc:1397] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260809 06:18:57.376869 5380 tablet_bootstrap.cc:492] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: No bootstrap required, opened a new log
I20260809 06:18:57.377110 5380 ts_tablet_manager.cc:1397] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 06:18:57.384352 5379 raft_consensus.cc:348] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.384702 5379 raft_consensus.cc:374] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.384830 5379 raft_consensus.cc:729] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8bd2bc9dc243f295a3b5f797e072cd, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.385416 5379 consensus_queue.cc:260] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.385823 5378 raft_consensus.cc:348] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.386226 5378 raft_consensus.cc:374] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.386330 5378 raft_consensus.cc:729] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aae5c624cca4755b82d7dba5ee44b57, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.386567 5213 heartbeater.cc:503] Master 127.3.137.126:34465 was elected leader, sending a full tablet report...
I20260809 06:18:57.386924 5378 consensus_queue.cc:260] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.387810 5379 ts_tablet_manager.cc:1428] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 06:18:57.387569 5380 raft_consensus.cc:348] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.388015 5380 raft_consensus.cc:374] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.388123 5380 raft_consensus.cc:729] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4859abb7c7420e8d6ee4a67c8149f4, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.388695 5380 consensus_queue.cc:260] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.389703 5358 heartbeater.cc:503] Master 127.3.137.126:34465 was elected leader, sending a full tablet report...
I20260809 06:18:57.390426 5380 ts_tablet_manager.cc:1428] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260809 06:18:57.397030 5378 ts_tablet_manager.cc:1428] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Time spent starting tablet: real 0.022s user 0.014s sys 0.004s
W20260809 06:18:57.443333 5067 tablet.cc:2369] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:18:57.473567 5214 tablet.cc:2369] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:18:57.496701 5359 tablet.cc:2369] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:18:57.578289 5384 raft_consensus.cc:482] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:18:57.578620 5384 raft_consensus.cc:504] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.580797 5384 leader_election.cc:288] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583), da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195)
I20260809 06:18:57.588335 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.581811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:60995 (local address 127.3.137.65:42583)
0809 06:18:57.582032 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.582038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.582059 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:18:57.582243 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.582249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.582335 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.582423 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.583391 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.584202 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.586004 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.587319 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.587486 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.587557 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.587932 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.588071 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:18:57.588388 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.581165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:42583 (local address 127.3.137.66:60995)
0809 06:18:57.581881 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.581903 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.582042 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.582682 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.582686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.582715 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.583201 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.583217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.584350 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.584353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.585835 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.585846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.587539 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.587546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.587733 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.587769 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.587863 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.587997 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":22,"spinlock_wait_cycles":28032,"thread_start_us":900,"threads_started":1}
I20260809 06:18:57.589442 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.581613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58865 (local address 127.3.137.67:33195)
0809 06:18:57.581978 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.581987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.582794 (+ 807us) server_negotiation.cc:407] Connection header received
0809 06:18:57.583123 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.583146 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.583233 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.583323 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.584113 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.584829 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.587014 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.588228 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.588398 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.588470 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.589053 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.589190 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260809 06:18:57.589418 5011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10cdc849a2249f585673c30613e19be" candidate_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aae5c624cca4755b82d7dba5ee44b57" is_pre_election: true
I20260809 06:18:57.590029 5011 raft_consensus.cc:2454] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8bd2bc9dc243f295a3b5f797e072cd in term 0.
I20260809 06:18:57.590198 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.581245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33195 (local address 127.3.137.66:58865)
0809 06:18:57.582691 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.582708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.582904 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.583490 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.583495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.583522 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.583920 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.583932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.584973 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.584980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.586789 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.586804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.588650 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.588659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.588838 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.588876 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.589607 (+ 731us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.589815 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":2,"thread_start_us":1054,"threads_started":1}
I20260809 06:18:57.591325 5092 leader_election.cc:302] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8bd2bc9dc243f295a3b5f797e072cd, 7aae5c624cca4755b82d7dba5ee44b57; no voters:
I20260809 06:18:57.591400 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10cdc849a2249f585673c30613e19be" candidate_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" is_pre_election: true
I20260809 06:18:57.591718 5384 raft_consensus.cc:2781] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:18:57.591876 5384 raft_consensus.cc:482] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:18:57.591986 5384 raft_consensus.cc:3037] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.591997 5312 raft_consensus.cc:2454] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8bd2bc9dc243f295a3b5f797e072cd in term 0.
I20260809 06:18:57.594412 5384 raft_consensus.cc:504] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.595058 5384 leader_election.cc:288] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [CANDIDATE]: Term 1 election: Requested vote from peers 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583), da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195)
I20260809 06:18:57.595482 5011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10cdc849a2249f585673c30613e19be" candidate_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aae5c624cca4755b82d7dba5ee44b57"
I20260809 06:18:57.595726 5011 raft_consensus.cc:3037] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.595721 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10cdc849a2249f585673c30613e19be" candidate_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4859abb7c7420e8d6ee4a67c8149f4"
I20260809 06:18:57.595923 5312 raft_consensus.cc:3037] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.599056 5011 raft_consensus.cc:2454] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8bd2bc9dc243f295a3b5f797e072cd in term 1.
I20260809 06:18:57.599086 5312 raft_consensus.cc:2454] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8bd2bc9dc243f295a3b5f797e072cd in term 1.
I20260809 06:18:57.599539 5088 leader_election.cc:302] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8bd2bc9dc243f295a3b5f797e072cd, da4859abb7c7420e8d6ee4a67c8149f4; no voters:
I20260809 06:18:57.599843 5384 raft_consensus.cc:2781] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:57.600301 5384 raft_consensus.cc:686] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 LEADER]: Becoming Leader. State: Replica: 0b8bd2bc9dc243f295a3b5f797e072cd, State: Running, Role: LEADER
I20260809 06:18:57.600833 5384 consensus_queue.cc:237] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } }
I20260809 06:18:57.606011 4839 catalog_manager.cc:5495] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66). New cstate: current_term: 1 leader_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } health_report { overall_health: HEALTHY } } }
I20260809 06:18:57.645243 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.639946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.3.137.66:36797)
0809 06:18:57.640133 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.640137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.640301 (+ 164us) server_negotiation.cc:407] Connection header received
0809 06:18:57.640600 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.640604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.640679 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.640798 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:18:57.641196 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.641986 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.643339 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.643733 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.643825 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.644286 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:18:57.644744 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:18:57.644851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.644989 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.645050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:18:57.652808 5312 raft_consensus.cc:1261] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Refusing update from remote peer 0b8bd2bc9dc243f295a3b5f797e072cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:18:57.652909 5011 raft_consensus.cc:1261] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Refusing update from remote peer 0b8bd2bc9dc243f295a3b5f797e072cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:18:57.653707 5389 consensus_queue.cc:1035] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:18:57.654083 5384 consensus_queue.cc:1035] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:18:57.668216 5394 mvcc.cc:205] Tried to move back new op lower bound from 7316505959017218048 to 7316505958817431552. Current Snapshot: MvccSnapshot[applied={T|T < 7316505959017218048}]
I20260809 06:18:57.686722 4840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:18:57.687059 4840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:18:57.691071 4840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Sending DeleteTablet for 3 replicas of tablet d10cdc849a2249f585673c30613e19be
I20260809 06:18:57.692257 5286 tablet_service.cc:1514] Processing DeleteTablet for tablet d10cdc849a2249f585673c30613e19be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:33598
I20260809 06:18:57.692257 4989 tablet_service.cc:1514] Processing DeleteTablet for tablet d10cdc849a2249f585673c30613e19be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:52788
I20260809 06:18:57.692322 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet d10cdc849a2249f585673c30613e19be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:46780
I20260809 06:18:57.695310 4840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:18:57.695987 4840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:18:57.704756 5139 tablet_service.cc:1467] Processing CreateTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 (DEFAULT_TABLE table=test-table [id=6ca62def9c6145b79458d09130349783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.705981 5139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5088a5b2674b66bb5bc6d36fa4a481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.705901 4988 tablet_service.cc:1467] Processing CreateTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 (DEFAULT_TABLE table=test-table [id=6ca62def9c6145b79458d09130349783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.706529 4988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5088a5b2674b66bb5bc6d36fa4a481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.706519 5401 tablet_replica.cc:330] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: stopping tablet replica
I20260809 06:18:57.707228 5401 raft_consensus.cc:2229] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:18:57.707883 5401 raft_consensus.cc:2258] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:57.709230 5286 tablet_service.cc:1467] Processing CreateTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 (DEFAULT_TABLE table=test-table [id=6ca62def9c6145b79458d09130349783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:18:57.709738 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5088a5b2674b66bb5bc6d36fa4a481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:57.713605 5378 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Bootstrap starting.
I20260809 06:18:57.715444 5378 tablet_bootstrap.cc:654] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.716626 5379 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Bootstrap starting.
I20260809 06:18:57.717540 5402 tablet_replica.cc:330] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: stopping tablet replica
I20260809 06:18:57.717747 5401 ts_tablet_manager.cc:1905] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:18:57.717991 5402 raft_consensus.cc:2229] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:18:57.718490 5402 raft_consensus.cc:2258] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:57.718954 5379 tablet_bootstrap.cc:654] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.719657 5380 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Bootstrap starting.
I20260809 06:18:57.721498 5380 tablet_bootstrap.cc:654] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:57.729027 5402 ts_tablet_manager.cc:1905] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:18:57.737227 5401 ts_tablet_manager.cc:1918] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:18:57.737705 5401 log.cc:1196] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/d10cdc849a2249f585673c30613e19be
I20260809 06:18:57.738339 5401 ts_tablet_manager.cc:1939] T d10cdc849a2249f585673c30613e19be P da4859abb7c7420e8d6ee4a67c8149f4: Deleting consensus metadata
I20260809 06:18:57.738395 5379 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: No bootstrap required, opened a new log
I20260809 06:18:57.738586 5379 ts_tablet_manager.cc:1397] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20260809 06:18:57.739074 5380 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: No bootstrap required, opened a new log
I20260809 06:18:57.739303 5380 ts_tablet_manager.cc:1397] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.010s
I20260809 06:18:57.739562 5379 raft_consensus.cc:348] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.739794 5379 raft_consensus.cc:374] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.739894 5379 raft_consensus.cc:729] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8bd2bc9dc243f295a3b5f797e072cd, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.740144 5380 raft_consensus.cc:348] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.740197 5379 consensus_queue.cc:260] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.740394 5380 raft_consensus.cc:374] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.740506 5380 raft_consensus.cc:729] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4859abb7c7420e8d6ee4a67c8149f4, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.740586 4824 catalog_manager.cc:4841] TS da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195): tablet d10cdc849a2249f585673c30613e19be (table test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]) successfully deleted
I20260809 06:18:57.740765 5380 consensus_queue.cc:260] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.741518 5379 ts_tablet_manager.cc:1428] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 06:18:57.741576 5403 tablet_replica.cc:330] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: stopping tablet replica
I20260809 06:18:57.744161 5380 ts_tablet_manager.cc:1428] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260809 06:18:57.744814 5403 raft_consensus.cc:2229] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:18:57.745476 5403 raft_consensus.cc:2258] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:57.746706 5378 tablet_bootstrap.cc:492] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: No bootstrap required, opened a new log
I20260809 06:18:57.746855 5378 ts_tablet_manager.cc:1397] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.002s
I20260809 06:18:57.747712 5378 raft_consensus.cc:348] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.747984 5378 raft_consensus.cc:374] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:57.748158 5378 raft_consensus.cc:729] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aae5c624cca4755b82d7dba5ee44b57, State: Initialized, Role: FOLLOWER
I20260809 06:18:57.748442 5378 consensus_queue.cc:260] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.749046 5378 ts_tablet_manager.cc:1428] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 06:18:57.749722 5403 ts_tablet_manager.cc:1905] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:18:57.755227 5402 ts_tablet_manager.cc:1918] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:18:57.755446 5402 log.cc:1196] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/d10cdc849a2249f585673c30613e19be
I20260809 06:18:57.756137 5402 ts_tablet_manager.cc:1939] T d10cdc849a2249f585673c30613e19be P 7aae5c624cca4755b82d7dba5ee44b57: Deleting consensus metadata
I20260809 06:18:57.757596 4827 catalog_manager.cc:4841] TS 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583): tablet d10cdc849a2249f585673c30613e19be (table test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]) successfully deleted
I20260809 06:18:57.758265 5403 ts_tablet_manager.cc:1918] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:18:57.758430 5403 log.cc:1196] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/d10cdc849a2249f585673c30613e19be
I20260809 06:18:57.759331 5403 ts_tablet_manager.cc:1939] T d10cdc849a2249f585673c30613e19be P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting consensus metadata
I20260809 06:18:57.761101 4825 catalog_manager.cc:4841] TS 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797): tablet d10cdc849a2249f585673c30613e19be (table test-table [id=2ed07aeedbfd454fa02b82c6a3e3df22]) successfully deleted
I20260809 06:18:57.927294 5385 raft_consensus.cc:482] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:18:57.927567 5385 raft_consensus.cc:504] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.929759 5385 leader_election.cc:288] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797), da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195)
I20260809 06:18:57.938429 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.930270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:52425 (local address 127.3.137.67:33195)
0809 06:18:57.930530 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.930539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.931218 (+ 679us) server_negotiation.cc:407] Connection header received
0809 06:18:57.931481 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.931488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.931583 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.931649 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.932501 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.933565 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.936077 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.937379 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.937556 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.937638 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.938001 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.938146 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:18:57.938535 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.930100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33195 (local address 127.3.137.65:52425)
0809 06:18:57.931064 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.931095 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.931252 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.931883 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.931887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.931913 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.932324 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.932337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.933721 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.933727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.935912 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.935926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.937610 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.937618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.937821 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.937856 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.937949 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.938092 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":41,"thread_start_us":568,"threads_started":1}
I20260809 06:18:57.939121 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.930156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:59959 (local address 127.3.137.66:36797)
0809 06:18:57.930378 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:57.930386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:57.931806 (+ 1420us) server_negotiation.cc:407] Connection header received
0809 06:18:57.931988 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:57.931994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:57.932139 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:57.932253 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:57.932911 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.934210 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.936553 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.937582 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.937754 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.937878 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:57.938305 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 06:18:57.938447 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:18:57.939355 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5088a5b2674b66bb5bc6d36fa4a481" candidate_uuid: "7aae5c624cca4755b82d7dba5ee44b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" is_pre_election: true
I20260809 06:18:57.939631 5312 raft_consensus.cc:2454] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aae5c624cca4755b82d7dba5ee44b57 in term 0.
I20260809 06:18:57.939924 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:57.930932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:36797 (local address 127.3.137.65:59959)
0809 06:18:57.931687 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:57.931701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:57.931849 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:57.932305 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:57.932310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:57.932334 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:57.932724 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.932737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.934372 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.934377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.936397 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:57.936407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:57.937876 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:57.937884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:57.938080 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:57.938116 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:18:57.939385 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0809 06:18:57.939531 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":29,"thread_start_us":436,"threads_started":1}
I20260809 06:18:57.940112 4934 leader_election.cc:302] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aae5c624cca4755b82d7dba5ee44b57, da4859abb7c7420e8d6ee4a67c8149f4; no voters:
I20260809 06:18:57.940877 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5088a5b2674b66bb5bc6d36fa4a481" candidate_uuid: "7aae5c624cca4755b82d7dba5ee44b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" is_pre_election: true
I20260809 06:18:57.941390 5162 raft_consensus.cc:2454] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aae5c624cca4755b82d7dba5ee44b57 in term 0.
I20260809 06:18:57.941712 5385 raft_consensus.cc:2781] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:18:57.941875 5385 raft_consensus.cc:482] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:18:57.941977 5385 raft_consensus.cc:3037] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.945232 5385 raft_consensus.cc:504] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.945952 5385 leader_election.cc:288] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [CANDIDATE]: Term 1 election: Requested vote from peers 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797), da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195)
I20260809 06:18:57.946386 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5088a5b2674b66bb5bc6d36fa4a481" candidate_uuid: "7aae5c624cca4755b82d7dba5ee44b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd"
I20260809 06:18:57.946616 5162 raft_consensus.cc:3037] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.946560 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5088a5b2674b66bb5bc6d36fa4a481" candidate_uuid: "7aae5c624cca4755b82d7dba5ee44b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4859abb7c7420e8d6ee4a67c8149f4"
I20260809 06:18:57.946774 5312 raft_consensus.cc:3037] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:57.949697 5312 raft_consensus.cc:2454] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aae5c624cca4755b82d7dba5ee44b57 in term 1.
I20260809 06:18:57.950062 5162 raft_consensus.cc:2454] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aae5c624cca4755b82d7dba5ee44b57 in term 1.
I20260809 06:18:57.950201 4934 leader_election.cc:302] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aae5c624cca4755b82d7dba5ee44b57, da4859abb7c7420e8d6ee4a67c8149f4; no voters:
I20260809 06:18:57.950495 5385 raft_consensus.cc:2781] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:57.951035 5385 raft_consensus.cc:686] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 LEADER]: Becoming Leader. State: Replica: 7aae5c624cca4755b82d7dba5ee44b57, State: Running, Role: LEADER
I20260809 06:18:57.951479 5385 consensus_queue.cc:237] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:57.955410 4840 catalog_manager.cc:5495] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "7aae5c624cca4755b82d7dba5ee44b57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:18:58 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:18:58.067365 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.049507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.3.137.126:34465)
0809 06:18:58.051927 (+ 2420us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.051931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.051951 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:18:58.051996 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.051999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.052051 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.052139 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.053666 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.054525 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.056428 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.056830 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.056924 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.061230 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.061235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.061238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.061268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.064046 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.064959 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.064963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.064965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.065079 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.065530 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.065533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.065535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.065968 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.066088 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.066683 (+ 595us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.067076 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":2325,"thread_start_us":2186,"threads_started":1}
I20260809 06:18:58.067394 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.049453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:34465 (local address 127.0.0.1:55716)
0809 06:18:58.051585 (+ 2132us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.051609 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.051784 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.052248 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.052254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.053214 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:18:58.053500 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.053512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.054670 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.054677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.056287 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.056303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.057022 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.057034 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.057219 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.057988 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:18:58.058022 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:18:58.060990 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:18:58.064190 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:58.064203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:58.064211 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:58.064835 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:58.065200 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:18:58.065207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:18:58.065212 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:18:58.065409 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:18:58.066138 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:18:58.066147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:18:58.066364 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.066584 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.066900 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1942,"thread_start_us":1747,"threads_started":1}
I20260809 06:18:58.073558 4840 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:18:58.075101 4840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:18:58.082621 4988 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6ca740e06c4a619735e69e3de61d9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f48d38ead99b49bc9dbbbf3b196c4631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:18:58.082926 5286 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6ca740e06c4a619735e69e3de61d9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f48d38ead99b49bc9dbbbf3b196c4631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:18:58.083338 4988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6ca740e06c4a619735e69e3de61d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:58.083549 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6ca740e06c4a619735e69e3de61d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:58.088289 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6ca740e06c4a619735e69e3de61d9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f48d38ead99b49bc9dbbbf3b196c4631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:18:58.089020 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6ca740e06c4a619735e69e3de61d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:18:58.089742 5380 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: Bootstrap starting.
I20260809 06:18:58.091074 5378 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: Bootstrap starting.
I20260809 06:18:58.091604 5380 tablet_bootstrap.cc:654] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:58.092835 5378 tablet_bootstrap.cc:654] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:58.100561 5379 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: Bootstrap starting.
I20260809 06:18:58.101586 5380 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: No bootstrap required, opened a new log
I20260809 06:18:58.101805 5380 ts_tablet_manager.cc:1397] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260809 06:18:58.101766 5378 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: No bootstrap required, opened a new log
I20260809 06:18:58.101994 5378 ts_tablet_manager.cc:1397] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260809 06:18:58.102350 5379 tablet_bootstrap.cc:654] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: Neither blocks nor log segments found. Creating new log.
I20260809 06:18:58.102777 5380 raft_consensus.cc:348] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.103123 5380 raft_consensus.cc:374] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:58.103238 5380 raft_consensus.cc:729] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4859abb7c7420e8d6ee4a67c8149f4, State: Initialized, Role: FOLLOWER
I20260809 06:18:58.103495 5378 raft_consensus.cc:348] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.103756 5378 raft_consensus.cc:374] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:58.103883 5378 raft_consensus.cc:729] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aae5c624cca4755b82d7dba5ee44b57, State: Initialized, Role: FOLLOWER
I20260809 06:18:58.103649 5380 consensus_queue.cc:260] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.104164 5378 consensus_queue.cc:260] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.104787 5378 ts_tablet_manager.cc:1428] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 06:18:58.104796 5385 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.106083 5385 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.104431 5386 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.107223 5386 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.111402 5380 ts_tablet_manager.cc:1428] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260809 06:18:58.116472 5379 tablet_bootstrap.cc:492] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: No bootstrap required, opened a new log
I20260809 06:18:58.116624 5379 ts_tablet_manager.cc:1397] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.014s
I20260809 06:18:58.117229 5379 raft_consensus.cc:348] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.117437 5379 raft_consensus.cc:374] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:18:58.117497 5379 raft_consensus.cc:729] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8bd2bc9dc243f295a3b5f797e072cd, State: Initialized, Role: FOLLOWER
I20260809 06:18:58.117697 5379 consensus_queue.cc:260] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.118265 5379 ts_tablet_manager.cc:1428] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:18:58.118111 5389 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.118467 5389 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.368744 5385 consensus_queue.cc:1035] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:18:58.375384 5427 consensus_queue.cc:1035] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:18:58.421363 5386 raft_consensus.cc:482] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:18:58.421583 5386 raft_consensus.cc:504] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.423207 5386 leader_election.cc:288] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583), 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797)
I20260809 06:18:58.431083 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.423700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:48771 (local address 127.3.137.66:36797)
0809 06:18:58.423940 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.423949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.425439 (+ 1490us) server_negotiation.cc:407] Connection header received
0809 06:18:58.425643 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.425649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.425739 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.425846 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:58.426357 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.427157 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.429325 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.430122 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.430308 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.430387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.430635 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.430784 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:18:58.431113 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.423448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:36797 (local address 127.3.137.67:48771)
0809 06:18:58.425185 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.425217 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.425427 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.425875 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.425879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.425906 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:58.426191 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.426199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.427310 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.427320 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.429159 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.429171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.430305 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.430311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.430475 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.430498 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.430609 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.430738 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":547,"threads_started":1}
I20260809 06:18:58.431779 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6ca740e06c4a619735e69e3de61d9d" candidate_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" is_pre_election: true
I20260809 06:18:58.432063 5162 raft_consensus.cc:2454] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da4859abb7c7420e8d6ee4a67c8149f4 in term 0.
I20260809 06:18:58.432317 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.423448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:42583 (local address 127.3.137.67:57681)
0809 06:18:58.425585 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.425606 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.425727 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.426179 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.426183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.426208 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:18:58.426572 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.426581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.428150 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.428157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.430122 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.430134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.431470 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.431476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.431625 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.431647 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.431760 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.431886 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"mutex_wait_us":4,"spinlock_wait_cycles":8960,"thread_start_us":1769,"threads_started":1}
I20260809 06:18:58.432610 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.423701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:57681 (local address 127.3.137.65:42583)
0809 06:18:58.425897 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.425901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.425915 (+ 14us) server_negotiation.cc:407] Connection header received
0809 06:18:58.425971 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.425974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.426032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.426149 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:18:58.426739 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.428016 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.430285 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.431204 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.431397 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.431490 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.431786 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.431920 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2049,"thread_start_us":1873,"threads_started":1}
I20260809 06:18:58.432613 5237 leader_election.cc:302] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8bd2bc9dc243f295a3b5f797e072cd, da4859abb7c7420e8d6ee4a67c8149f4; no voters:
I20260809 06:18:58.433220 5011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6ca740e06c4a619735e69e3de61d9d" candidate_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aae5c624cca4755b82d7dba5ee44b57" is_pre_election: true
I20260809 06:18:58.433365 5386 raft_consensus.cc:2781] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:18:58.433516 5386 raft_consensus.cc:482] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:18:58.433516 5011 raft_consensus.cc:2454] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da4859abb7c7420e8d6ee4a67c8149f4 in term 0.
I20260809 06:18:58.433610 5386 raft_consensus.cc:3037] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:58.436379 5386 raft_consensus.cc:504] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.436967 5386 leader_election.cc:288] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [CANDIDATE]: Term 1 election: Requested vote from peers 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583), 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797)
I20260809 06:18:58.437513 5011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6ca740e06c4a619735e69e3de61d9d" candidate_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aae5c624cca4755b82d7dba5ee44b57"
I20260809 06:18:58.437520 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6ca740e06c4a619735e69e3de61d9d" candidate_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd"
I20260809 06:18:58.437695 5011 raft_consensus.cc:3037] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:58.437788 5162 raft_consensus.cc:3037] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:18:58.440284 5011 raft_consensus.cc:2454] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da4859abb7c7420e8d6ee4a67c8149f4 in term 1.
I20260809 06:18:58.440582 5162 raft_consensus.cc:2454] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da4859abb7c7420e8d6ee4a67c8149f4 in term 1.
I20260809 06:18:58.441278 5239 leader_election.cc:302] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aae5c624cca4755b82d7dba5ee44b57, da4859abb7c7420e8d6ee4a67c8149f4; no voters:
I20260809 06:18:58.441767 5386 raft_consensus.cc:2781] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:18:58.442263 5386 raft_consensus.cc:686] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 LEADER]: Becoming Leader. State: Replica: da4859abb7c7420e8d6ee4a67c8149f4, State: Running, Role: LEADER
I20260809 06:18:58.442703 5386 consensus_queue.cc:237] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } }
I20260809 06:18:58.444765 5434 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: TxnStatusTablet state changed. Reason: New leader da4859abb7c7420e8d6ee4a67c8149f4. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.445065 5434 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:18:58.446333 5437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:18:58.446235 4840 catalog_manager.cc:5495] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67). New cstate: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } health_report { overall_health: HEALTHY } } }
I20260809 06:18:58.446633 5437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:18:58.478359 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.470303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33195 (local address 127.0.0.1:53964)
0809 06:18:58.470639 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.470661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.470841 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.471396 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.471403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.471419 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:18:58.471622 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.471634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.473057 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.473063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.475280 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.475296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.476052 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.476059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.476256 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.476309 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.477368 (+ 1059us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.477391 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.477479 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.477639 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260809 06:18:58.478529 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.469895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.3.137.67:33195)
0809 06:18:58.470617 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.470628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.470747 (+ 119us) server_negotiation.cc:407] Connection header received
0809 06:18:58.471098 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.471102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.471187 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.471335 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:18:58.471776 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.472882 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.475420 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.475875 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.476018 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.476548 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.477240 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.477414 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.477524 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.477615 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":384,"threads_started":1}
I20260809 06:18:58.483222 5011 raft_consensus.cc:1261] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Refusing update from remote peer da4859abb7c7420e8d6ee4a67c8149f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:18:58.483258 5162 raft_consensus.cc:1261] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Refusing update from remote peer da4859abb7c7420e8d6ee4a67c8149f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:18:58.484506 5434 consensus_queue.cc:1035] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 06:18:58.484889 5386 consensus_queue.cc:1035] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:18:58.487862 5427 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: TxnStatusTablet state changed. Reason: New leader da4859abb7c7420e8d6ee4a67c8149f4. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.488147 5427 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.491747 5426 mvcc.cc:205] Tried to move back new op lower bound from 7316505962419314688 to 7316505962264694784. Current Snapshot: MvccSnapshot[applied={T|T < 7316505962419314688}]
I20260809 06:18:58.492652 5386 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.492975 5386 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:18:58.494318 5427 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.494661 5427 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 7aae5c624cca4755b82d7dba5ee44b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.499991 5389 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: TxnStatusTablet state changed. Reason: New leader da4859abb7c7420e8d6ee4a67c8149f4. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.500687 5389 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.502118 5434 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.502404 5434 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P da4859abb7c7420e8d6ee4a67c8149f4: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:18:58.504259 5429 mvcc.cc:205] Tried to move back new op lower bound from 7316505962419314688 to 7316505962264694784. Current Snapshot: MvccSnapshot[applied={T|T < 7316505962419314688}]
I20260809 06:18:58.506925 5389 tablet_replica.cc:439] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aae5c624cca4755b82d7dba5ee44b57" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42583 } } peers { permanent_uuid: "0b8bd2bc9dc243f295a3b5f797e072cd" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36797 } } peers { permanent_uuid: "da4859abb7c7420e8d6ee4a67c8149f4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33195 } } }
I20260809 06:18:58.507222 5389 tablet_replica.cc:442] T 5b6ca740e06c4a619735e69e3de61d9d P 0b8bd2bc9dc243f295a3b5f797e072cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:18:58.516469 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.506531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.3.137.65:42583)
0809 06:18:58.506728 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.506738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.510437 (+ 3699us) server_negotiation.cc:407] Connection header received
0809 06:18:58.510694 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.510698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.510777 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.510955 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:18:58.511700 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.512750 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.514214 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.514642 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.514794 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.515317 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.515855 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.516022 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.516110 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.516194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 06:18:58.534494 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.528197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:45019 (local address 127.3.137.67:33195)
0809 06:18:58.528399 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.528404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.529148 (+ 744us) server_negotiation.cc:407] Connection header received
0809 06:18:58.529368 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.529374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.529450 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.529561 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:18:58.530014 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.530943 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.532614 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.533158 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.533296 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.533734 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.533833 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.533942 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.534086 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.534160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:18:58.534559 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.528202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33195 (local address 127.3.137.65:45019)
0809 06:18:58.529028 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.529043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.529225 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.529571 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.529576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.529587 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:18:58.529864 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.529873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.531120 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.531125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.532445 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.532485 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.533318 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.533324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.533477 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.533538 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.533943 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.533955 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.534054 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.534183 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":421,"threads_started":1}
I20260809 06:18:58.577742 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.568949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:49135 (local address 127.3.137.65:42583)
0809 06:18:58.569145 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.569153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.570499 (+ 1346us) server_negotiation.cc:407] Connection header received
0809 06:18:58.570820 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.570827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.570913 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.571080 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:18:58.572055 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.573199 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.575160 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.576177 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.576332 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.576862 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.577032 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.577125 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.577351 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.577426 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:18:58.577849 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.569027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:42583 (local address 127.3.137.67:49135)
0809 06:18:58.570323 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0809 06:18:58.570420 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0809 06:18:58.570641 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:18:58.571304 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:18:58.571308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:18:58.571321 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:18:58.571871 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.571885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.573341 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.573345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.575009 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:18:58.575025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.576393 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.576400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:18:58.576622 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.576676 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:18:58.577174 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:18:58.577185 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 06:18:58.577289 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 06:18:58.577423 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":649,"threads_started":1}
I20260809 06:18:58.633440 4839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:18:58.633745 4839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:18:58.637410 4839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80368557fe8c4d50a532b71ab9f7d3a3: Sending DeleteTablet for 3 replicas of tablet ab5088a5b2674b66bb5bc6d36fa4a481
I20260809 06:18:58.638280 4988 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:58 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:52788
I20260809 06:18:58.638473 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:58 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:46780
I20260809 06:18:58.639297 5286 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5088a5b2674b66bb5bc6d36fa4a481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:18:58 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:33598
I20260809 06:18:58.649984 5456 tablet_replica.cc:330] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: stopping tablet replica
I20260809 06:18:58.650489 5456 raft_consensus.cc:2229] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:18:58.651233 5456 raft_consensus.cc:2258] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:58.652505 5456 ts_tablet_manager.cc:1905] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:18:58.662555 5456 ts_tablet_manager.cc:1918] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:18:58.662782 5456 log.cc:1196] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/ab5088a5b2674b66bb5bc6d36fa4a481
I20260809 06:18:58.663429 5456 ts_tablet_manager.cc:1939] T ab5088a5b2674b66bb5bc6d36fa4a481 P 7aae5c624cca4755b82d7dba5ee44b57: Deleting consensus metadata
I20260809 06:18:58.664696 4827 catalog_manager.cc:4841] TS 7aae5c624cca4755b82d7dba5ee44b57 (127.3.137.65:42583): tablet ab5088a5b2674b66bb5bc6d36fa4a481 (table test-table [id=6ca62def9c6145b79458d09130349783]) successfully deleted
I20260809 06:18:58.668308 5458 tablet_replica.cc:330] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: stopping tablet replica
I20260809 06:18:58.668725 5458 raft_consensus.cc:2229] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:18:58.669169 5458 raft_consensus.cc:2258] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:58.670600 5458 ts_tablet_manager.cc:1905] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:18:58.671211 5457 tablet_replica.cc:330] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: stopping tablet replica
I20260809 06:18:58.677821 5457 raft_consensus.cc:2229] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:18:58.678136 5457 raft_consensus.cc:2258] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:18:58.679620 5457 ts_tablet_manager.cc:1905] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:18:58 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:18:58.687251 5458 ts_tablet_manager.cc:1918] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:18:58.690311 5457 ts_tablet_manager.cc:1918] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:18:58.690551 5458 log.cc:1196] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/ab5088a5b2674b66bb5bc6d36fa4a481
I20260809 06:18:58.691174 5458 ts_tablet_manager.cc:1939] T ab5088a5b2674b66bb5bc6d36fa4a481 P da4859abb7c7420e8d6ee4a67c8149f4: Deleting consensus metadata
I20260809 06:18:58.692730 4824 catalog_manager.cc:4841] TS da4859abb7c7420e8d6ee4a67c8149f4 (127.3.137.67:33195): tablet ab5088a5b2674b66bb5bc6d36fa4a481 (table test-table [id=6ca62def9c6145b79458d09130349783]) successfully deleted
I20260809 06:18:58.693050 5457 log.cc:1196] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/ab5088a5b2674b66bb5bc6d36fa4a481
I20260809 06:18:58.693573 5457 ts_tablet_manager.cc:1939] T ab5088a5b2674b66bb5bc6d36fa4a481 P 0b8bd2bc9dc243f295a3b5f797e072cd: Deleting consensus metadata
I20260809 06:18:58.694831 4825 catalog_manager.cc:4841] TS 0b8bd2bc9dc243f295a3b5f797e072cd (127.3.137.66:36797): tablet ab5088a5b2674b66bb5bc6d36fa4a481 (table test-table [id=6ca62def9c6145b79458d09130349783]) successfully deleted
I20260809 06:18:58.695086 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.668327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.3.137.65:42583)
0809 06:18:58.668508 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.668514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.672089 (+ 3575us) server_negotiation.cc:407] Connection header received
0809 06:18:58.672613 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.672619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.672702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.672792 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.675435 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.676791 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.678902 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.679739 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.679848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.685801 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.685820 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.685871 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.685942 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.690547 (+ 4605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.691618 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.691628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.691632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.691808 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.692294 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.692300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.692305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.693099 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.693421 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.693934 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.694170 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 06:18:58.696761 5060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44212
I20260809 06:18:58.732210 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.700808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.3.137.126:34465)
0809 06:18:58.713459 (+ 12651us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.713465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.714130 (+ 665us) server_negotiation.cc:407] Connection header received
0809 06:18:58.714293 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.714299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.714395 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.714514 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.716475 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.717728 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.719612 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.720290 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.720414 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.724135 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.724141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.724146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.724185 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.728301 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.729137 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.729143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.729147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.729318 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.729776 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.729782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.729785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.730134 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.730382 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.731580 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.731815 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12506,"thread_start_us":12382,"threads_started":1}
W20260809 06:18:58.733184 4839 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60872
I20260809 06:18:58.735670 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4918
I20260809 06:18:58.767783 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.736875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.3.137.65:42583)
0809 06:18:58.737065 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.737071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.750201 (+ 13130us) server_negotiation.cc:407] Connection header received
0809 06:18:58.750407 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.750413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.750547 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.750640 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.752444 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.753694 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.755645 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.756507 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.756600 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.759796 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.759802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.759807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.759852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.763915 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.764710 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.764716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.764719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.764993 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.765464 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.765469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.765472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.765820 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.766069 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.767176 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.767400 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 06:18:58.768539 5060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44228
W20260809 06:18:58.769121 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:18:58.769296 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4918
I20260809 06:18:58.791256 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5070
Aug 09 06:18:58 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:18:58.808208 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.792085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.3.137.66:36797)
0809 06:18:58.792276 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.792282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.792307 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:18:58.792509 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.792516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.792599 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.792714 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.793807 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.794914 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.796302 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.796979 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.797120 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.801144 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.801162 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.801179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.801220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.805177 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.805977 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.805988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.805991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.806209 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.806599 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.806603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.806608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.807083 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.807430 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.807578 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.807841 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 06:18:58.809027 5208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52812
W20260809 06:18:58.809739 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:18:58.809938 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5070
I20260809 06:18:58.838126 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5217
Aug 09 06:18:58 dist-test-slave-2w3w krb5kdc[4795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:18:58.856920 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:18:58.839050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.3.137.67:33195)
0809 06:18:58.839303 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 06:18:58.839310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:18:58.839362 (+ 52us) server_negotiation.cc:407] Connection header received
0809 06:18:58.839554 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:18:58.839560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:18:58.839678 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:18:58.839818 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:18:58.841256 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.842456 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.843948 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:18:58.844633 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:18:58.844782 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:18:58.849556 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:18:58.849570 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:18:58.849590 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:18:58.849636 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:18:58.853848 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.854604 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.854614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.854618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.854800 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:18:58.855288 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:18:58.855293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:18:58.855297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:18:58.855710 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:18:58.856030 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 06:18:58.856289 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 06:18:58.856515 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 06:18:58.857676 5353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53966
W20260809 06:18:58.858453 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:18:58.858609 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5217
I20260809 06:18:58.878921 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 4808
I20260809 06:18:58.880170 4883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:18:58.997550 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 4808
2026-08-09T06:18:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5838 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:18:59 dist-test-slave-2w3w krb5kdc[5468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:18:59 dist-test-slave-2w3w krb5kdc[5468](info): set up 2 sockets
Aug 09 06:18:59 dist-test-slave-2w3w krb5kdc[5468](info): commencing operation
krb5kdc: starting...
W20260809 06:19:00.125927 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:00 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:00.330320 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:44515
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:00.656525 5482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:00.656821 5482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:00.656912 5482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:00.664180 5482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:00.664288 5482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:19:00.664330 5482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:00.664386 5482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:00.664438 5482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:00.664501 5482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:00.674786 5482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:44515
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:00.676124 5482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:00.677455 5482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:00.689322 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:00.689209 5491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:00.689335 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:00.690191 5482 server_base.cc:1034] running on GCE node
I20260809 06:19:00.690886 5482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:00.692278 5482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:00.693512 5482 hybrid_clock.cc:648] HybridClock initialized: now 1786256340693438 us; error 35 us; skew 500 ppm
I20260809 06:19:00.693871 5482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:00 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:00.701800 5482 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:00.703922 5482 webserver.cc:460] Webserver started at http://127.3.137.126:35177/ using document root <none> and password file <none>
I20260809 06:19:00.704902 5482 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:00.705029 5482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:00.705335 5482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:00.708539 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "697d40517f5b4d30abe863fe015aa2a1"
format_stamp: "Formatted at 2026-08-09 06:19:00 on dist-test-slave-2w3w"
server_key: "f8a96a355af8c58e32bf1e4c4648eca8"
server_key_iv: "7ad670d86c4602e568f691acdacf0f3b"
server_key_version: "encryptionkey@0"
I20260809 06:19:00.709308 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "697d40517f5b4d30abe863fe015aa2a1"
format_stamp: "Formatted at 2026-08-09 06:19:00 on dist-test-slave-2w3w"
server_key: "f8a96a355af8c58e32bf1e4c4648eca8"
server_key_iv: "7ad670d86c4602e568f691acdacf0f3b"
server_key_version: "encryptionkey@0"
I20260809 06:19:00.714597 5482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:19:00.718103 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:00.719448 5482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:00.719626 5482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "697d40517f5b4d30abe863fe015aa2a1"
format_stamp: "Formatted at 2026-08-09 06:19:00 on dist-test-slave-2w3w"
server_key: "f8a96a355af8c58e32bf1e4c4648eca8"
server_key_iv: "7ad670d86c4602e568f691acdacf0f3b"
server_key_version: "encryptionkey@0"
I20260809 06:19:00.719838 5482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:00.738312 5482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:00.742158 5482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:00.742416 5482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:00.765377 5482 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:44515
I20260809 06:19:00.765384 5564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:44515 every 8 connection(s)
I20260809 06:19:00.766834 5482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:00.768544 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5482
I20260809 06:19:00.768774 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:00.769476 3621 external_mini_cluster.cc:1428] Setting key d283401f70d2efa4189534666c62c682
I20260809 06:19:00.770941 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:19:00 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:00.784957 5565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Bootstrap starting.
I20260809 06:19:00.789232 5565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:00.790505 5565 log.cc:824] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:00.793876 5565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: No bootstrap required, opened a new log
I20260809 06:19:00.794129 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:00.771507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.3.137.126:44515)
0809 06:19:00.772371 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:00.772386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:00.772437 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:19:00.774335 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:00.774362 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:00.774895 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:00.775345 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:00.776656 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:00.778799 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:00.780154 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:00.781198 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:00.781397 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:00.785597 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:00.785608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:00.785626 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:00.785676 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:00.789727 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:00.790402 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:00.790411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:00.790418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:00.790628 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:00.791047 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:00.791055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:00.791061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:00.791665 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:00.791958 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:00.792395 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 06:19:00.792816 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":382,"threads_started":1}
I20260809 06:19:00.801980 5565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } }
I20260809 06:19:00.802314 5565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:00.802415 5565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697d40517f5b4d30abe863fe015aa2a1, State: Initialized, Role: FOLLOWER
I20260809 06:19:00.802940 5565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } }
I20260809 06:19:00.803233 5565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:00.803412 5565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:00.803589 5565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:00.805927 5565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } }
I20260809 06:19:00.806421 5565 leader_election.cc:302] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 697d40517f5b4d30abe863fe015aa2a1; no voters:
I20260809 06:19:00.807073 5565 leader_election.cc:288] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:00.807214 5570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:00.807919 5570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [term 1 LEADER]: Becoming Leader. State: Replica: 697d40517f5b4d30abe863fe015aa2a1, State: Running, Role: LEADER
I20260809 06:19:00.808414 5570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } }
I20260809 06:19:00.809095 5565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:00.811612 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "697d40517f5b4d30abe863fe015aa2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } } }
I20260809 06:19:00.812006 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:00.812772 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697d40517f5b4d30abe863fe015aa2a1. Latest consensus state: current_term: 1 leader_uuid: "697d40517f5b4d30abe863fe015aa2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697d40517f5b4d30abe863fe015aa2a1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44515 } } }
I20260809 06:19:00.813047 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:00.815932 5575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:00.821210 5575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:00.829917 5575 catalog_manager.cc:1347] Generated new cluster ID: 6ee0ace1f0594e8282cde0e7699a78cb
I20260809 06:19:00.830063 5575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:00.846145 5575 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:00.848090 5575 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:00.867151 5575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Generated new TSK 0
I20260809 06:19:00.868036 5575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:00.916476 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:44515
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:01.243956 5591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:01.244376 5591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:01.244562 5591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:01.251940 5591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:01.252086 5591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:01.252254 5591 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:19:01.252341 5591 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:01.252441 5591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:01.261884 5591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:01.263301 5591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:01.264609 5591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:01.277271 5596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:01.277402 5599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:01.277462 5597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:01.278306 5591 server_base.cc:1034] running on GCE node
I20260809 06:19:01.278904 5591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:01.279873 5591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:01.281066 5591 hybrid_clock.cc:648] HybridClock initialized: now 1786256341281017 us; error 52 us; skew 500 ppm
I20260809 06:19:01.281373 5591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:01 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:01.284615 5591 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:01.286075 5591 webserver.cc:460] Webserver started at http://127.3.137.65:35587/ using document root <none> and password file <none>
I20260809 06:19:01.286721 5591 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:01.286880 5591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:01.287297 5591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:01.290444 5591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "f8f9243c4978480c8986d4691b96e0d9"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "a2db72f51477ca8f67702a950f5dd329"
server_key_iv: "d3d9a9009a1be6a275c36ccf93bb5e16"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.291301 5591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "f8f9243c4978480c8986d4691b96e0d9"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "a2db72f51477ca8f67702a950f5dd329"
server_key_iv: "d3d9a9009a1be6a275c36ccf93bb5e16"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.296689 5591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:01.300048 5606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:01.301278 5591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:01.301469 5591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "f8f9243c4978480c8986d4691b96e0d9"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "a2db72f51477ca8f67702a950f5dd329"
server_key_iv: "d3d9a9009a1be6a275c36ccf93bb5e16"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.301676 5591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:01.315256 5591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:01.318941 5591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:01.319294 5591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:01.321919 5591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:01.322052 5591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:01.322180 5591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:01.322257 5591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:01 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:01.346475 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.322338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:43457 (local address 127.3.137.126:44515)
0809 06:19:01.323519 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:01.323528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:01.324105 (+ 577us) server_negotiation.cc:407] Connection header received
0809 06:19:01.326342 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:01.326348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:01.326436 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:01.326562 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:01.328725 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.329809 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.332568 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.333151 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.333298 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.338637 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:01.338643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:01.338650 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:01.338693 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:01.342562 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.343805 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.343813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.343819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.344026 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.344687 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.344693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.344696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.345006 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:01.345178 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:01.345857 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 06:19:01.346090 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":845,"threads_started":1}
I20260809 06:19:01.347671 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.323033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.65:43457)
0809 06:19:01.323941 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:01.323978 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:01.326017 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:01.326804 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:01.326811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:01.327603 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:01.328518 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.328549 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.330012 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.330020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.332384 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.332408 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.333423 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.333430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.333643 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.334527 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:01.334557 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:01.338286 (+ 3729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:01.342760 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.342773 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.342792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.343612 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.344185 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.344192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.344198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.344506 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.345154 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:01.345161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:01.345488 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 06:19:01.345751 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 06:19:01.346243 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":448,"threads_started":1}
I20260809 06:19:01.374543 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.358217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:33051 (local address 127.3.137.126:44515)
0809 06:19:01.358469 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:01.358478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:01.359968 (+ 1490us) server_negotiation.cc:407] Connection header received
0809 06:19:01.360189 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:01.360192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:01.360260 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:01.360341 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:01.361887 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.362584 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.365255 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.365882 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.366036 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.368384 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:01.368391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:01.368412 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:01.368453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:01.371615 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.372583 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.372589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.372592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.372731 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.373130 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.373134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.373139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.373394 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:01.373551 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:01.373957 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 06:19:01.374175 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:19:01.374640 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.358108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.65:33051)
0809 06:19:01.359836 (+ 1728us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:01.359860 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:01.360054 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:01.360414 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:01.360422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:01.361185 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:01.361652 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.361669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.362733 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.362741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.365036 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.365062 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.366111 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.366118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.366282 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.366913 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:01.366930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:01.368205 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:01.371759 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.371764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.371768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.372447 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.372862 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.372867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.372871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.373000 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.373517 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:01.373521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:01.373699 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 06:19:01.373845 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 06:19:01.374170 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"thread_start_us":1169,"threads_started":1}
I20260809 06:19:01.381978 5591 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:44649
I20260809 06:19:01.382025 5737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:44649 every 8 connection(s)
I20260809 06:19:01.383198 5591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:01.385146 5738 heartbeater.cc:347] Connected to a master server at 127.3.137.126:44515
I20260809 06:19:01.385403 5738 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:01.385743 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5591
I20260809 06:19:01.385933 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:01.386148 5738 heartbeater.cc:511] Master 127.3.137.126:44515 requested a full tablet report, sending...
I20260809 06:19:01.386695 3621 external_mini_cluster.cc:1428] Setting key 88f158df3e5de0a54d5a00bf2577f903
I20260809 06:19:01.388376 5516 ts_manager.cc:194] Registered new tserver with Master: f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649)
I20260809 06:19:01.389909 5516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:43457
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:01.432669 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:44515
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:01.755267 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:01.755563 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:01.755750 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:01.762657 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:01.762852 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:01.763036 5742 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:19:01.763141 5742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:01.763247 5742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:01.777597 5742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:01.779083 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:01.780361 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:01.793368 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:01.793467 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:01.793432 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:01.794590 5742 server_base.cc:1034] running on GCE node
I20260809 06:19:01.795367 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:01.796226 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:01.797446 5742 hybrid_clock.cc:648] HybridClock initialized: now 1786256341797397 us; error 44 us; skew 500 ppm
I20260809 06:19:01.797767 5742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:01 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:01.801086 5742 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:01.802667 5742 webserver.cc:460] Webserver started at http://127.3.137.66:35037/ using document root <none> and password file <none>
I20260809 06:19:01.803426 5742 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:01.803563 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:01.803926 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:01.807176 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "d0df4ba3684efa38ce96145ee3bc4020"
server_key_iv: "25bc8514c57b30beccfd1b19f9e7a724"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.807999 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "d0df4ba3684efa38ce96145ee3bc4020"
server_key_iv: "25bc8514c57b30beccfd1b19f9e7a724"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.813740 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:19:01.817559 5759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:01.818858 5742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:19:01.819056 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
format_stamp: "Formatted at 2026-08-09 06:19:01 on dist-test-slave-2w3w"
server_key: "d0df4ba3684efa38ce96145ee3bc4020"
server_key_iv: "25bc8514c57b30beccfd1b19f9e7a724"
server_key_version: "encryptionkey@0"
I20260809 06:19:01.819249 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:01.850238 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:01.854041 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:01.854333 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:01.857167 5742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:01.857302 5742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:01.857427 5742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:01.857507 5742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:01 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:01.881763 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.857566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:56775 (local address 127.3.137.126:44515)
0809 06:19:01.857758 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:01.857766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:01.859223 (+ 1457us) server_negotiation.cc:407] Connection header received
0809 06:19:01.861416 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:01.861426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:01.861532 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:01.861676 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:01.863876 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.865143 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.867860 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.868300 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.868467 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.874154 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:01.874162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:01.874169 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:01.874214 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:01.877599 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.878657 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.878665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.878669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.878833 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.879548 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.879557 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.879563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.880027 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:01.880208 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:01.880894 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 06:19:01.881201 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:19:01.882299 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.858198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.66:56775)
0809 06:19:01.859058 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:01.859096 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:01.860913 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:01.861956 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:01.861967 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:01.862760 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:01.863627 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.863678 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.865347 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.865350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.867708 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.867721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.868568 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.868581 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.868829 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.869592 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:01.869615 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:01.873872 (+ 4257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:01.877784 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.877794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.877805 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.878474 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.878995 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.879000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.879006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.879370 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.880174 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:01.880179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:01.880475 (+ 296us) client_negotiation.cc:769] Sending connection context
0809 06:19:01.880712 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 06:19:01.881067 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":363,"threads_started":1}
I20260809 06:19:01.904083 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.889042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39687 (local address 127.3.137.126:44515)
0809 06:19:01.889255 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:01.889263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:01.890250 (+ 987us) server_negotiation.cc:407] Connection header received
0809 06:19:01.890368 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:01.890375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:01.890468 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:01.890577 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:01.892053 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.892784 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.894109 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.894515 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.894663 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.897354 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:01.897360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:01.897367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:01.897404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:01.900574 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.901505 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.901509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.901511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.901636 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:01.902056 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:01.902059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:01.902061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:01.902519 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:01.902694 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:01.903590 (+ 896us) server_negotiation.cc:299] Negotiation successful
0809 06:19:01.903838 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:19:01.904281 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:01.889170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.66:39687)
0809 06:19:01.890144 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:01.890161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:01.890267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:01.890636 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:01.890643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:01.891281 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:01.891852 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.891865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.892924 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.892931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.893962 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:01.893974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:01.894704 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:01.894729 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:01.894905 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:01.895652 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:01.895694 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:01.897143 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:01.900741 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.900749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.900756 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.901382 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.901755 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:01.901763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:01.901769 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:01.901935 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:01.902659 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:01.902668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:01.902920 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 06:19:01.903149 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 06:19:01.903577 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":626,"threads_started":1}
I20260809 06:19:01.913276 5742 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:38519
I20260809 06:19:01.913311 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:38519 every 8 connection(s)
I20260809 06:19:01.914573 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:01.916718 5887 heartbeater.cc:347] Connected to a master server at 127.3.137.126:44515
I20260809 06:19:01.916965 5887 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:01.917577 5887 heartbeater.cc:511] Master 127.3.137.126:44515 requested a full tablet report, sending...
I20260809 06:19:01.918889 5516 ts_manager.cc:194] Registered new tserver with Master: 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519)
I20260809 06:19:01.919719 5516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:56775
I20260809 06:19:01.922441 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5742
I20260809 06:19:01.922629 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:01.923209 3621 external_mini_cluster.cc:1428] Setting key faf561894264d012e4bc3e74c9966a0a
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:01.968493 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:44515
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:02.287902 5891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:02.288147 5891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:02.288246 5891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:02.295152 5891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:02.295250 5891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:02.295338 5891 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:19:02.295392 5891 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:02.295463 5891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:02.304838 5891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:02.306106 5891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:02.307649 5891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:02.320493 5900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:02.320608 5897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:02.320571 5898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:02.321491 5891 server_base.cc:1034] running on GCE node
I20260809 06:19:02.322161 5891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:02.323401 5891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:02.324608 5891 hybrid_clock.cc:648] HybridClock initialized: now 1786256342324554 us; error 50 us; skew 500 ppm
I20260809 06:19:02.325023 5891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:02 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:02.328460 5891 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:02.329840 5891 webserver.cc:460] Webserver started at http://127.3.137.67:41639/ using document root <none> and password file <none>
I20260809 06:19:02.330503 5891 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:02.330662 5891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:02.331063 5891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:02.333952 5891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "05351bd1156c453c80115b5ff101c8be"
format_stamp: "Formatted at 2026-08-09 06:19:02 on dist-test-slave-2w3w"
server_key: "74c0ec49074d44b85b7e7fff8e596765"
server_key_iv: "5ba04a73ebbbb76941dbbc7379b4aea8"
server_key_version: "encryptionkey@0"
I20260809 06:19:02.334795 5891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "05351bd1156c453c80115b5ff101c8be"
format_stamp: "Formatted at 2026-08-09 06:19:02 on dist-test-slave-2w3w"
server_key: "74c0ec49074d44b85b7e7fff8e596765"
server_key_iv: "5ba04a73ebbbb76941dbbc7379b4aea8"
server_key_version: "encryptionkey@0"
I20260809 06:19:02.339927 5891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:19:02.343169 5908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:02.344419 5891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:19:02.344609 5891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "05351bd1156c453c80115b5ff101c8be"
format_stamp: "Formatted at 2026-08-09 06:19:02 on dist-test-slave-2w3w"
server_key: "74c0ec49074d44b85b7e7fff8e596765"
server_key_iv: "5ba04a73ebbbb76941dbbc7379b4aea8"
server_key_version: "encryptionkey@0"
I20260809 06:19:02.344813 5891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:02.361557 5891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:02.365833 5891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:02.366127 5891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:02.368805 5891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:02.368960 5891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:02.369098 5891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:02.369205 5891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:02 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:02.391196 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.369606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:42997 (local address 127.3.137.126:44515)
0809 06:19:02.369799 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.369804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.371442 (+ 1638us) server_negotiation.cc:407] Connection header received
0809 06:19:02.373512 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.373518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.373610 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.373733 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:02.376528 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.377314 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.379932 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.380454 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.380586 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.384946 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:02.384951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:02.384954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:02.385009 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:02.388043 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.388874 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.388884 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.388886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.389008 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.389459 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.389463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.389465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.389725 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:02.389919 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.390587 (+ 668us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.390823 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:19:02.392426 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.370335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.67:42997)
0809 06:19:02.371248 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.371310 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.373168 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.374167 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.374181 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.375286 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:02.376348 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.376367 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.377509 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.377514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.379793 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.379805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.380931 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.380944 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.381375 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.382084 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:02.382105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:02.384627 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:02.388188 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.388196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.388209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.388746 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.389123 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.389128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.389132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.389330 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.389870 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:02.389880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:02.390198 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.390452 (+ 254us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.390934 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":404,"threads_started":1}
I20260809 06:19:02.392531 5738 heartbeater.cc:503] Master 127.3.137.126:44515 was elected leader, sending a full tablet report...
I20260809 06:19:02.413812 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.398638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:42223 (local address 127.3.137.126:44515)
0809 06:19:02.398862 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.398870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.399578 (+ 708us) server_negotiation.cc:407] Connection header received
0809 06:19:02.399762 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.399765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.399833 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.399944 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:02.401398 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.402449 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.404483 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.405082 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.405238 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.407797 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:02.407803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:02.407808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:02.407849 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:02.411162 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.411805 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.411810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.411814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.411939 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.412312 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.412317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.412320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.412589 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:02.412757 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.413205 (+ 448us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.413434 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:02.413992 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.398635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.3.137.67:42223)
0809 06:19:02.399454 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.399480 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.399643 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.399976 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.399981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.400728 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:02.401180 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.401199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.402591 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.402600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.404336 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.404351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.405541 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.405547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.405808 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.406489 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:02.406506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:02.407554 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:02.411297 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.411304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.411310 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.411680 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.412050 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.412055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.412059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.412190 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.412708 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:02.412714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:02.412901 (+ 187us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.413062 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.413417 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":429,"threads_started":1}
I20260809 06:19:02.428990 5891 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:38979
I20260809 06:19:02.429031 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:38979 every 8 connection(s)
I20260809 06:19:02.430260 5891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:02.432327 6034 heartbeater.cc:347] Connected to a master server at 127.3.137.126:44515
I20260809 06:19:02.432570 6034 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:02.433245 6034 heartbeater.cc:511] Master 127.3.137.126:44515 requested a full tablet report, sending...
I20260809 06:19:02.434629 5516 ts_manager.cc:194] Registered new tserver with Master: 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:02.435552 5516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:42997
I20260809 06:19:02.438221 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 5891
I20260809 06:19:02.438398 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:02.438908 3621 external_mini_cluster.cc:1428] Setting key 5eeac6632d676e92715455d5a4734d4f
I20260809 06:19:02.441754 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:19:02 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:19:02 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:02.492053 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.477999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.3.137.126:44515)
0809 06:19:02.478196 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.478201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.478543 (+ 342us) server_negotiation.cc:407] Connection header received
0809 06:19:02.478698 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.478702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.478767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.478888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:02.480342 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.481144 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.482358 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.482769 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.482873 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.486851 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:02.486856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:02.486859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:02.486892 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:02.489749 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.490340 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.490344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.490346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.490464 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.490808 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.490812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.490814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.491167 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:02.491357 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.491604 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.491765 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:19:02.496311 5516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:02.503187 5516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:02.535856 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.516401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.0.0.1:39944)
0809 06:19:02.527340 (+ 10939us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.527363 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.527500 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.528328 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.528336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.528380 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:02.528819 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.528834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.531021 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.531028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.533171 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.533183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.534802 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.534810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.535068 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.535119 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.535292 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.535464 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10793,"thread_start_us":10636,"threads_started":1}
I20260809 06:19:02.535871 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.517026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.3.137.67:38979)
0809 06:19:02.527790 (+ 10764us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.527797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.527825 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:19:02.527907 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.527912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.528100 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.528396 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:02.529080 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.530832 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.533349 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.534565 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.534753 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.535030 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.535373 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.535535 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10604,"thread_start_us":10401,"threads_started":1}
I20260809 06:19:02.537415 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.516548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.3.137.65:44649)
0809 06:19:02.529540 (+ 12992us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.529549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.529579 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:19:02.529679 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.529688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.529910 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.530133 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:02.531096 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.532274 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.534850 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.536311 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.536475 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.536677 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.536938 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.537094 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12828,"thread_start_us":12680,"threads_started":1}
I20260809 06:19:02.537945 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.516367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44649 (local address 127.0.0.1:47488)
0809 06:19:02.526353 (+ 9986us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.526410 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.526662 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.530177 (+ 3515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.530185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.530225 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:02.530801 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.530813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.532421 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.532426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.534669 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.534683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.536539 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.536546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.536728 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.536760 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.536885 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.537016 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9565,"thread_start_us":14069,"threads_started":1}
I20260809 06:19:02.540500 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.517045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38519 (local address 127.0.0.1:35622)
0809 06:19:02.530722 (+ 13677us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.530744 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.530981 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.531839 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.531844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.531868 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:02.532294 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.532307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.534179 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.534184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.537692 (+ 3508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.537708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.539745 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.539751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.539916 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.539946 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.540046 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.540164 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13546,"thread_start_us":8697,"threads_started":1}
I20260809 06:19:02.540578 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.517741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.3.137.66:38519)
0809 06:19:02.529091 (+ 11350us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.529099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.530850 (+ 1751us) server_negotiation.cc:407] Connection header received
0809 06:19:02.531205 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.531213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.531422 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.531583 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:02.532555 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.533999 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.537872 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.539228 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.539403 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.539589 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.540127 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.540290 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11182,"thread_start_us":11027,"threads_started":1}
I20260809 06:19:02.540330 5966 tablet_service.cc:1467] Processing CreateTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d (DEFAULT_TABLE table=test-table [id=52df1c135926456cb67af722295e9a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:02.541785 5966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccbb29ff9444eaa849d19f5dac6f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.541658 5660 tablet_service.cc:1467] Processing CreateTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d (DEFAULT_TABLE table=test-table [id=52df1c135926456cb67af722295e9a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:02.542836 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccbb29ff9444eaa849d19f5dac6f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.544173 5813 tablet_service.cc:1467] Processing CreateTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d (DEFAULT_TABLE table=test-table [id=52df1c135926456cb67af722295e9a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:02.545440 5813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdccbb29ff9444eaa849d19f5dac6f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.555120 6055 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Bootstrap starting.
I20260809 06:19:02.556435 6057 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Bootstrap starting.
I20260809 06:19:02.559152 6055 tablet_bootstrap.cc:654] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.559396 6058 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Bootstrap starting.
I20260809 06:19:02.560086 6057 tablet_bootstrap.cc:654] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.560669 6055 log.cc:824] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:02.561386 6057 log.cc:824] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:02.562664 6058 tablet_bootstrap.cc:654] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.563723 6058 log.cc:824] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:02.564982 6055 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: No bootstrap required, opened a new log
I20260809 06:19:02.565280 6055 ts_tablet_manager.cc:1397] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260809 06:19:02.566397 6057 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: No bootstrap required, opened a new log
I20260809 06:19:02.566649 6057 ts_tablet_manager.cc:1397] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 06:19:02.566679 6058 tablet_bootstrap.cc:492] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: No bootstrap required, opened a new log
I20260809 06:19:02.566924 6058 ts_tablet_manager.cc:1397] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260809 06:19:02.574047 6057 raft_consensus.cc:348] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.574450 6057 raft_consensus.cc:374] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.574580 6057 raft_consensus.cc:729] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.575383 6057 consensus_queue.cc:260] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.576071 6055 raft_consensus.cc:348] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.576510 6055 raft_consensus.cc:374] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.576550 6034 heartbeater.cc:503] Master 127.3.137.126:44515 was elected leader, sending a full tablet report...
I20260809 06:19:02.576614 6055 raft_consensus.cc:729] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.577207 6055 consensus_queue.cc:260] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.577541 6058 raft_consensus.cc:348] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.578014 6058 raft_consensus.cc:374] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.578172 6058 raft_consensus.cc:729] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.578622 6057 ts_tablet_manager.cc:1428] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260809 06:19:02.578755 6058 consensus_queue.cc:260] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.580127 6055 ts_tablet_manager.cc:1428] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260809 06:19:02.580989 5887 heartbeater.cc:503] Master 127.3.137.126:44515 was elected leader, sending a full tablet report...
I20260809 06:19:02.581581 6058 ts_tablet_manager.cc:1428] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260809 06:19:02.635637 5739 tablet.cc:2369] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:02.640450 6065 raft_consensus.cc:482] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:02.640826 6065 raft_consensus.cc:504] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.642534 6065 leader_election.cc:288] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:02.652000 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.642986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:54949 (local address 127.3.137.67:38979)
0809 06:19:02.643183 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.643188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.644172 (+ 984us) server_negotiation.cc:407] Connection header received
0809 06:19:02.644381 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.644387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.644463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.644567 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:02.645460 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.646594 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.649459 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.650772 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.651019 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.651099 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.651602 (+ 503us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.651745 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:19:02.652000 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.642916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:55597 (local address 127.3.137.65:44649)
0809 06:19:02.643178 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.643184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.644259 (+ 1075us) server_negotiation.cc:407] Connection header received
0809 06:19:02.644380 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.644385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.644463 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.644568 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:02.645455 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.646607 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.649465 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.650791 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.650949 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.651050 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.651602 (+ 552us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.651744 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 06:19:02.652305 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.642884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44649 (local address 127.3.137.66:55597)
0809 06:19:02.644156 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.644171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.644264 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.644603 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.644609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.644652 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:02.645246 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.645266 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.646738 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.646743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.649306 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.649315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.651099 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.651104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.651361 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.651433 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.651537 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.651757 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":212,"spinlock_wait_cycles":9728,"thread_start_us":490,"threads_started":1}
I20260809 06:19:02.652333 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.642827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.3.137.66:54949)
0809 06:19:02.644045 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.644065 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.644232 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.644603 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.644609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.644644 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:02.645325 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.645338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.646740 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.646744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.649279 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.649301 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.651096 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.651103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.651367 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.651433 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.651537 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.651757 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":233,"spinlock_wait_cycles":2048,"thread_start_us":1011,"threads_started":1}
I20260809 06:19:02.653321 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccbb29ff9444eaa849d19f5dac6f4d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be" is_pre_election: true
I20260809 06:19:02.653434 5687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccbb29ff9444eaa849d19f5dac6f4d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9" is_pre_election: true
I20260809 06:19:02.653870 5987 raft_consensus.cc:2454] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 0.
I20260809 06:19:02.653945 5687 raft_consensus.cc:2454] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 0.
I20260809 06:19:02.654582 5761 leader_election.cc:302] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:02.655074 6065 raft_consensus.cc:2781] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:02.655224 6065 raft_consensus.cc:482] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:02.655331 6065 raft_consensus.cc:3037] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:02.657614 6065 raft_consensus.cc:504] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
I20260809 06:19:02.658166 6065 leader_election.cc:288] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 election: Requested vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:02.658638 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccbb29ff9444eaa849d19f5dac6f4d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be"
I20260809 06:19:02.658638 5687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdccbb29ff9444eaa849d19f5dac6f4d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9"
I20260809 06:19:02.658849 5687 raft_consensus.cc:3037] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:02.658849 5987 raft_consensus.cc:3037] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:02.662134 5687 raft_consensus.cc:2454] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 1.
I20260809 06:19:02.662138 5987 raft_consensus.cc:2454] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 1.
I20260809 06:19:02.662544 5760 leader_election.cc:302] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:02.662916 6065 raft_consensus.cc:2781] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:02.663403 6065 raft_consensus.cc:686] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 LEADER]: Becoming Leader. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Running, Role: LEADER
I20260809 06:19:02.663866 6065 consensus_queue.cc:237] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } }
W20260809 06:19:02.666720 5888 tablet.cc:2369] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:02.668582 5514 catalog_manager.cc:5495] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66). New cstate: current_term: 1 leader_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } health_report { overall_health: HEALTHY } } }
W20260809 06:19:02.681895 6035 tablet.cc:2369] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 06:19:02 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:02.726738 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.713255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44515 (local address 127.0.0.1:38656)
0809 06:19:02.713824 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:02.713845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:02.714008 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:02.714421 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:02.714424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:02.715291 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:02.715682 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.715698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.716706 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.716713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.717701 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:02.717712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.718473 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.718480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:02.718627 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.719303 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:02.719324 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:02.721409 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:02.724542 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.724553 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.724559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.724948 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.725281 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:02.725285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:02.725287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:02.725432 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:02.725911 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:02.725917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:02.726065 (+ 148us) client_negotiation.cc:769] Sending connection context
0809 06:19:02.726197 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 06:19:02.726395 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
I20260809 06:19:02.726871 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:02.713195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.3.137.126:44515)
0809 06:19:02.713375 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:02.713380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:02.713934 (+ 554us) server_negotiation.cc:407] Connection header received
0809 06:19:02.714180 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:02.714187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:02.714283 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:02.714397 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:02.715842 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.716581 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.717846 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:02.718280 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:02.718409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:02.721584 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:02.721597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:02.721601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:02.721628 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:02.724368 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.725053 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.725057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.725059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.725183 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:02.725534 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:02.725538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:02.725540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:02.725802 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:02.725908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:02.726299 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 06:19:02.726519 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":55}
I20260809 06:19:02.731760 5514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:38656:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:19:02.732821 5514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:02.739706 5813 tablet_service.cc:1467] Processing CreateTablet for tablet 1843e9c5f2454b36ab1e29fde44838fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98fba29723f6447d8bff6224e18f005c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:02.739603 5660 tablet_service.cc:1467] Processing CreateTablet for tablet 1843e9c5f2454b36ab1e29fde44838fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98fba29723f6447d8bff6224e18f005c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:02.739971 5966 tablet_service.cc:1467] Processing CreateTablet for tablet 1843e9c5f2454b36ab1e29fde44838fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98fba29723f6447d8bff6224e18f005c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:02.740542 5813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1843e9c5f2454b36ab1e29fde44838fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.740545 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1843e9c5f2454b36ab1e29fde44838fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.740792 5966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1843e9c5f2454b36ab1e29fde44838fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:02.746548 6057 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: Bootstrap starting.
I20260809 06:19:02.747587 6058 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Bootstrap starting.
I20260809 06:19:02.748237 6055 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: Bootstrap starting.
I20260809 06:19:02.748283 6057 tablet_bootstrap.cc:654] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.749397 6058 tablet_bootstrap.cc:654] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.750038 6055 tablet_bootstrap.cc:654] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:02.750824 6057 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: No bootstrap required, opened a new log
I20260809 06:19:02.751030 6057 ts_tablet_manager.cc:1397] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 06:19:02.751739 6058 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: No bootstrap required, opened a new log
I20260809 06:19:02.751893 6058 ts_tablet_manager.cc:1397] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:19:02.751861 6057 raft_consensus.cc:348] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.752122 6057 raft_consensus.cc:374] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.752285 6057 raft_consensus.cc:729] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.752656 6057 consensus_queue.cc:260] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.752713 6058 raft_consensus.cc:348] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.752936 6058 raft_consensus.cc:374] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.753055 6058 raft_consensus.cc:729] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.753394 6057 ts_tablet_manager.cc:1428] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 06:19:02.753278 6058 consensus_queue.cc:260] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.753854 6058 ts_tablet_manager.cc:1428] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 06:19:02.753811 6062 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:02.754196 6062 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:02.754526 6065 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:02.754907 6065 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:02.756083 6055 tablet_bootstrap.cc:492] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: No bootstrap required, opened a new log
I20260809 06:19:02.756264 6055 ts_tablet_manager.cc:1397] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260809 06:19:02.756968 6055 raft_consensus.cc:348] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.757297 6055 raft_consensus.cc:374] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:02.757436 6055 raft_consensus.cc:729] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Initialized, Role: FOLLOWER
I20260809 06:19:02.757691 6055 consensus_queue.cc:260] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:02.758268 6055 ts_tablet_manager.cc:1428] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 06:19:02.758201 6063 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:02.758610 6063 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:03.052835 6062 raft_consensus.cc:482] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:03.053146 6062 raft_consensus.cc:504] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.055334 6062 leader_election.cc:288] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519)
I20260809 06:19:03.061761 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.055711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:50001 (local address 127.3.137.65:44649)
0809 06:19:03.055939 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.055946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.056308 (+ 362us) server_negotiation.cc:407] Connection header received
0809 06:19:03.056542 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.056548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.056627 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.056677 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:03.057301 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.058068 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.060006 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.060772 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.060954 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.061034 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.061347 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.061475 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:03.061883 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.055619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44649 (local address 127.3.137.67:50001)
0809 06:19:03.056173 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.056191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.056351 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.056889 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.056892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.056912 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:03.057125 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.057132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.058219 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.058224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.059841 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.059857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.060955 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.060960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.061181 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.061204 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.061320 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.061458 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":7,"thread_start_us":285,"threads_started":1}
I20260809 06:19:03.062002 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.055645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38519 (local address 127.3.137.67:55025)
0809 06:19:03.056442 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.056474 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.056588 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.056977 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.056982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.057011 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:03.057330 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.057338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.058366 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.058372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.059949 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.059963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.061114 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.061123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.061275 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.061298 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.061404 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.061560 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":1,"spinlock_wait_cycles":15104,"thread_start_us":634,"threads_started":1}
I20260809 06:19:03.062237 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.056141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:55025 (local address 127.3.137.66:38519)
0809 06:19:03.056675 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.056682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.056698 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:19:03.056767 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.056770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.056824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.056963 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:03.057502 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.058221 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.060108 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.060872 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.061088 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.061159 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.061441 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.061574 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":281,"threads_started":1}
I20260809 06:19:03.062774 5687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1843e9c5f2454b36ab1e29fde44838fd" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9" is_pre_election: true
I20260809 06:19:03.063104 5687 raft_consensus.cc:2454] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 0.
I20260809 06:19:03.063573 5909 leader_election.cc:302] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:03.063525 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1843e9c5f2454b36ab1e29fde44838fd" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" is_pre_election: true
I20260809 06:19:03.063956 5839 raft_consensus.cc:2454] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 0.
I20260809 06:19:03.063987 6062 raft_consensus.cc:2781] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:03.064150 6062 raft_consensus.cc:482] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:03.064265 6062 raft_consensus.cc:3037] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.066598 6062 raft_consensus.cc:504] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.067283 6062 leader_election.cc:288] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 election: Requested vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519)
I20260809 06:19:03.067688 5687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1843e9c5f2454b36ab1e29fde44838fd" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9"
I20260809 06:19:03.067799 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1843e9c5f2454b36ab1e29fde44838fd" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
I20260809 06:19:03.067919 5687 raft_consensus.cc:3037] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.068019 5839 raft_consensus.cc:3037] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.070858 5687 raft_consensus.cc:2454] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 1.
I20260809 06:19:03.071000 5839 raft_consensus.cc:2454] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 1.
I20260809 06:19:03.071308 5909 leader_election.cc:302] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:03.071615 6062 raft_consensus.cc:2781] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:03.071966 6062 raft_consensus.cc:686] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [term 1 LEADER]: Becoming Leader. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Running, Role: LEADER
I20260809 06:19:03.072444 6062 consensus_queue.cc:237] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.076169 6084 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: TxnStatusTablet state changed. Reason: New leader 05351bd1156c453c80115b5ff101c8be. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.076519 6084 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:03.077189 6086 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:19:03.077090 5515 catalog_manager.cc:5495] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be reported cstate change: term changed from 0 to 1, leader changed from <none> to 05351bd1156c453c80115b5ff101c8be (127.3.137.67). New cstate: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } health_report { overall_health: HEALTHY } } }
I20260809 06:19:03.077446 6086 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:19:03.118608 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.112229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.0.0.1:39960)
0809 06:19:03.112470 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.112486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.112588 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.113160 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.113164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.113172 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:03.113347 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.113354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.115011 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.115018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.116192 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.116205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.116965 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.116972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.117138 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.117193 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.118030 (+ 837us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.118073 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.118179 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.118325 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 06:19:03.118649 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.112351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.3.137.67:38979)
0809 06:19:03.112528 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.112535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.112555 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:19:03.112839 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.112847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.112984 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.113140 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.113495 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.114823 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.116342 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.116786 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.116943 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.117393 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.117906 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.118083 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.118213 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.118297 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:19:03.126884 5687 raft_consensus.cc:1261] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Refusing update from remote peer 05351bd1156c453c80115b5ff101c8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:03.126940 5839 raft_consensus.cc:1261] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Refusing update from remote peer 05351bd1156c453c80115b5ff101c8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:03.127877 6084 consensus_queue.cc:1035] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.128269 6062 consensus_queue.cc:1035] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.138321 6065 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: TxnStatusTablet state changed. Reason: New leader 05351bd1156c453c80115b5ff101c8be. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.138883 6065 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:03.138885 6063 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: TxnStatusTablet state changed. Reason: New leader 05351bd1156c453c80115b5ff101c8be. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.139217 6063 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:03.142498 6089 mvcc.cc:205] Tried to move back new op lower bound from 7316505981437431808 to 7316505981228138496. Current Snapshot: MvccSnapshot[applied={T|T < 7316505981437431808}]
I20260809 06:19:03.143178 6084 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.143450 6084 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:03.145418 6062 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.145769 6062 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 05351bd1156c453c80115b5ff101c8be: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:03.147667 6063 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.147919 6063 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P f8f9243c4978480c8986d4691b96e0d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:03.152710 6069 consensus_queue.cc:1035] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.187604 6101 consensus_queue.cc:1035] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.191366 6065 tablet_replica.cc:439] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } } }
I20260809 06:19:03.191712 6065 tablet_replica.cc:442] T 1843e9c5f2454b36ab1e29fde44838fd P 4ed67d1bcc4b4b8ebd930e4f026f2bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:03.199311 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.169797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.3.137.66:38519)
0809 06:19:03.170041 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.170050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.182223 (+ 12173us) server_negotiation.cc:407] Connection header received
0809 06:19:03.182935 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.182945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.183130 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.183297 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.184331 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.185906 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.190436 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.191402 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.191530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.194884 (+ 3354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.195918 (+ 1034us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.196083 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.198902 (+ 2819us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.199019 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:03.216528 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.210773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:33413 (local address 127.3.137.67:38979)
0809 06:19:03.211016 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.211024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.211386 (+ 362us) server_negotiation.cc:407] Connection header received
0809 06:19:03.211603 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.211611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.211751 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.211881 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.212312 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.213068 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.214410 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.215095 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.215339 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.215809 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.215919 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.216032 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.216146 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.216223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 06:19:03.216572 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.210654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.3.137.66:33413)
0809 06:19:03.211282 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.211298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.211460 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.211892 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.211897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.211909 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:03.212158 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.212167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.213201 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.213208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.214257 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.214269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.215326 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.215332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.215490 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.215541 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.216033 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.216046 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.216123 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.216257 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
I20260809 06:19:03.256637 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.249448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38519 (local address 127.3.137.67:37707)
0809 06:19:03.250352 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.250433 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.250637 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.251028 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.251033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.251044 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:03.251431 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.251446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.252905 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.252909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.254340 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.254359 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.255532 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.255541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.255712 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.255771 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.256186 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.256203 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.256263 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.256372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":480,"threads_started":1}
I20260809 06:19:03.256633 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.249395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:37707 (local address 127.3.137.66:38519)
0809 06:19:03.249599 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.249605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.250523 (+ 918us) server_negotiation.cc:407] Connection header received
0809 06:19:03.250796 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.250802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.250887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.251015 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.251626 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.252768 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.254678 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.255375 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.255510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.255945 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.256081 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.256174 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.256291 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.256371 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:19:03.294198 5514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:03.294492 5514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:03.298046 5514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Sending DeleteTablet for 3 replicas of tablet fdccbb29ff9444eaa849d19f5dac6f4d
I20260809 06:19:03.299206 5660 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:03 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47488
I20260809 06:19:03.299453 5813 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:03 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:35622
I20260809 06:19:03.300285 5966 tablet_service.cc:1514] Processing DeleteTablet for tablet fdccbb29ff9444eaa849d19f5dac6f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:03 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:39944
I20260809 06:19:03.306989 5514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:03.307812 5514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:03.317456 6111 tablet_replica.cc:330] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: stopping tablet replica
I20260809 06:19:03.318611 6111 raft_consensus.cc:2229] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:03.318585 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 74631b175f73492ca4822f000b62d80d (DEFAULT_TABLE table=test-table [id=2b23996034024d658528c83ac3be3ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:03.319290 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74631b175f73492ca4822f000b62d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:03.319769 5812 tablet_service.cc:1467] Processing CreateTablet for tablet 74631b175f73492ca4822f000b62d80d (DEFAULT_TABLE table=test-table [id=2b23996034024d658528c83ac3be3ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:03.320003 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 74631b175f73492ca4822f000b62d80d (DEFAULT_TABLE table=test-table [id=2b23996034024d658528c83ac3be3ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:03.320315 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74631b175f73492ca4822f000b62d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:03.320551 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74631b175f73492ca4822f000b62d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:03.330075 6111 raft_consensus.cc:2258] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:03.332352 6113 tablet_replica.cc:330] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: stopping tablet replica
I20260809 06:19:03.332957 6113 raft_consensus.cc:2229] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:03.333391 6113 raft_consensus.cc:2258] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:03.334208 6112 tablet_replica.cc:330] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: stopping tablet replica
I20260809 06:19:03.333809 6111 ts_tablet_manager.cc:1905] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:03.335503 6112 raft_consensus.cc:2229] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:03.336637 6113 ts_tablet_manager.cc:1905] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:03.337421 6112 raft_consensus.cc:2258] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:03.337960 6116 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Bootstrap starting.
I20260809 06:19:03.342149 6115 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Bootstrap starting.
I20260809 06:19:03.344568 6112 ts_tablet_manager.cc:1905] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:03.345696 6115 tablet_bootstrap.cc:654] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:03.346210 6111 ts_tablet_manager.cc:1918] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:03.346467 6111 log.cc:1196] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/fdccbb29ff9444eaa849d19f5dac6f4d
I20260809 06:19:03.347172 6111 ts_tablet_manager.cc:1939] T fdccbb29ff9444eaa849d19f5dac6f4d P f8f9243c4978480c8986d4691b96e0d9: Deleting consensus metadata
I20260809 06:19:03.345121 6114 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Bootstrap starting.
I20260809 06:19:03.349156 5500 catalog_manager.cc:4841] TS f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649): tablet fdccbb29ff9444eaa849d19f5dac6f4d (table test-table [id=52df1c135926456cb67af722295e9a70]) successfully deleted
I20260809 06:19:03.346525 6116 tablet_bootstrap.cc:654] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:03.350921 6114 tablet_bootstrap.cc:654] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:03.357182 6116 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: No bootstrap required, opened a new log
I20260809 06:19:03.357370 6112 ts_tablet_manager.cc:1918] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:03.357630 6112 log.cc:1196] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/fdccbb29ff9444eaa849d19f5dac6f4d
I20260809 06:19:03.358373 6112 ts_tablet_manager.cc:1939] T fdccbb29ff9444eaa849d19f5dac6f4d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting consensus metadata
I20260809 06:19:03.358539 6114 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: No bootstrap required, opened a new log
I20260809 06:19:03.358695 6114 ts_tablet_manager.cc:1397] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260809 06:19:03.359563 6114 raft_consensus.cc:348] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.359853 6114 raft_consensus.cc:374] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:03.359956 6114 raft_consensus.cc:729] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Initialized, Role: FOLLOWER
I20260809 06:19:03.360198 6114 consensus_queue.cc:260] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.360585 5503 catalog_manager.cc:4841] TS 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519): tablet fdccbb29ff9444eaa849d19f5dac6f4d (table test-table [id=52df1c135926456cb67af722295e9a70]) successfully deleted
I20260809 06:19:03.363246 6116 ts_tablet_manager.cc:1397] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20260809 06:19:03.363958 6116 raft_consensus.cc:348] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.364174 6116 raft_consensus.cc:374] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:03.364303 6116 raft_consensus.cc:729] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Initialized, Role: FOLLOWER
I20260809 06:19:03.364574 6116 consensus_queue.cc:260] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.365167 6116 ts_tablet_manager.cc:1428] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:19:03.366279 6113 ts_tablet_manager.cc:1918] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:03.366487 6113 log.cc:1196] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/fdccbb29ff9444eaa849d19f5dac6f4d
I20260809 06:19:03.367123 6113 ts_tablet_manager.cc:1939] T fdccbb29ff9444eaa849d19f5dac6f4d P 05351bd1156c453c80115b5ff101c8be: Deleting consensus metadata
I20260809 06:19:03.369012 5501 catalog_manager.cc:4841] TS 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979): tablet fdccbb29ff9444eaa849d19f5dac6f4d (table test-table [id=52df1c135926456cb67af722295e9a70]) successfully deleted
I20260809 06:19:03.369753 6114 ts_tablet_manager.cc:1428] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260809 06:19:03.371397 6115 tablet_bootstrap.cc:492] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: No bootstrap required, opened a new log
I20260809 06:19:03.371578 6115 ts_tablet_manager.cc:1397] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.007s
I20260809 06:19:03.372357 6115 raft_consensus.cc:348] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.372653 6115 raft_consensus.cc:374] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:03.372763 6115 raft_consensus.cc:729] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Initialized, Role: FOLLOWER
I20260809 06:19:03.373028 6115 consensus_queue.cc:260] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.373734 6115 ts_tablet_manager.cc:1428] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 06:19:03.820128 6062 raft_consensus.cc:482] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:03.820392 6062 raft_consensus.cc:504] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.821017 6062 leader_election.cc:288] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519)
I20260809 06:19:03.821678 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74631b175f73492ca4822f000b62d80d" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9" is_pre_election: true
I20260809 06:19:03.821779 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74631b175f73492ca4822f000b62d80d" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" is_pre_election: true
I20260809 06:19:03.821941 5682 raft_consensus.cc:2454] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 0.
I20260809 06:19:03.822052 5839 raft_consensus.cc:2454] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 0.
I20260809 06:19:03.822525 5909 leader_election.cc:302] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:03.822847 6062 raft_consensus.cc:2781] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:03.823014 6062 raft_consensus.cc:482] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:03.823124 6062 raft_consensus.cc:3037] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.825398 6062 raft_consensus.cc:504] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.826069 6062 leader_election.cc:288] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 election: Requested vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519)
I20260809 06:19:03.826650 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74631b175f73492ca4822f000b62d80d" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
I20260809 06:19:03.826650 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74631b175f73492ca4822f000b62d80d" candidate_uuid: "05351bd1156c453c80115b5ff101c8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9"
I20260809 06:19:03.826928 5839 raft_consensus.cc:3037] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.826932 5682 raft_consensus.cc:3037] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:03.830165 5839 raft_consensus.cc:2454] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 1.
I20260809 06:19:03.830166 5682 raft_consensus.cc:2454] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05351bd1156c453c80115b5ff101c8be in term 1.
I20260809 06:19:03.830663 5913 leader_election.cc:302] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:03.831022 6062 raft_consensus.cc:2781] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:03.831434 6062 raft_consensus.cc:686] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 1 LEADER]: Becoming Leader. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Running, Role: LEADER
I20260809 06:19:03.831732 6062 consensus_queue.cc:237] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:03.834450 5514 catalog_manager.cc:5495] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be reported cstate change: term changed from 0 to 1, leader changed from <none> to 05351bd1156c453c80115b5ff101c8be (127.3.137.67). New cstate: current_term: 1 leader_uuid: "05351bd1156c453c80115b5ff101c8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } health_report { overall_health: HEALTHY } } }
I20260809 06:19:03.915884 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.910303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.3.137.67:38979)
0809 06:19:03.910754 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.910758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.910856 (+ 98us) server_negotiation.cc:407] Connection header received
0809 06:19:03.911050 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.911053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.911114 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.911196 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.911573 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.912280 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.913706 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.914339 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.914447 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.915025 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.915154 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.915243 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.915420 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.915502 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260809 06:19:03.934268 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.928467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:44573 (local address 127.3.137.67:38979)
0809 06:19:03.928727 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:03.928733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:03.929060 (+ 327us) server_negotiation.cc:407] Connection header received
0809 06:19:03.929224 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:03.929228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:03.929290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:03.929365 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:03.929910 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.930641 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.932620 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.933051 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.933149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.933616 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.933717 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.933831 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:03.933973 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 06:19:03.934036 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 06:19:03.934828 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:03.928354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.3.137.67:44573)
0809 06:19:03.928943 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:03.928972 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:03.929109 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:03.929404 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:03.929408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:03.929420 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:03.929664 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.929673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.930771 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.930777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.932475 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:03.932489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:03.933213 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:03.933219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:03.933399 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:03.933447 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:03.933850 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:03.933862 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 06:19:03.933938 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 06:19:03.934092 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":286,"threads_started":1}
I20260809 06:19:03.943562 5839 raft_consensus.cc:1261] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Refusing update from remote peer 05351bd1156c453c80115b5ff101c8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:03.943828 5682 raft_consensus.cc:1261] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Refusing update from remote peer 05351bd1156c453c80115b5ff101c8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:03.944352 6130 consensus_queue.cc:1035] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.944667 6120 consensus_queue.cc:1035] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:03.951615 6127 mvcc.cc:205] Tried to move back new op lower bound from 7316505984788062208 to 7316505984336932864. Current Snapshot: MvccSnapshot[applied={T|T < 7316505984788062208}]
W20260809 06:19:03.967047 6125 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 06:19:05.561066 5514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:05.561299 5514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:05.565490 5514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Sending DeleteTablet for 3 replicas of tablet 74631b175f73492ca4822f000b62d80d
I20260809 06:19:05.566504 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet 74631b175f73492ca4822f000b62d80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47488
I20260809 06:19:05.566562 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet 74631b175f73492ca4822f000b62d80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:39944
I20260809 06:19:05.566634 5812 tablet_service.cc:1514] Processing DeleteTablet for tablet 74631b175f73492ca4822f000b62d80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:35622
I20260809 06:19:05.579093 6166 tablet_replica.cc:330] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: stopping tablet replica
I20260809 06:19:05.579233 6168 tablet_replica.cc:330] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: stopping tablet replica
I20260809 06:19:05.579509 6166 raft_consensus.cc:2229] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:05.579509 6168 raft_consensus.cc:2229] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:05.579900 6166 raft_consensus.cc:2258] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:05.579924 6168 raft_consensus.cc:2258] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:05.581103 6168 ts_tablet_manager.cc:1905] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.581103 6166 ts_tablet_manager.cc:1905] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.589003 6168 ts_tablet_manager.cc:1918] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:05.589003 6166 ts_tablet_manager.cc:1918] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:05.589157 6166 log.cc:1196] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/74631b175f73492ca4822f000b62d80d
I20260809 06:19:05.589243 6168 log.cc:1196] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/74631b175f73492ca4822f000b62d80d
I20260809 06:19:05.589797 6166 ts_tablet_manager.cc:1939] T 74631b175f73492ca4822f000b62d80d P f8f9243c4978480c8986d4691b96e0d9: Deleting consensus metadata
I20260809 06:19:05.589833 6168 ts_tablet_manager.cc:1939] T 74631b175f73492ca4822f000b62d80d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting consensus metadata
I20260809 06:19:05.591169 5503 catalog_manager.cc:4841] TS 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519): tablet 74631b175f73492ca4822f000b62d80d (table test-table [id=2b23996034024d658528c83ac3be3ec1]) successfully deleted
I20260809 06:19:05.591169 5500 catalog_manager.cc:4841] TS f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649): tablet 74631b175f73492ca4822f000b62d80d (table test-table [id=2b23996034024d658528c83ac3be3ec1]) successfully deleted
I20260809 06:19:05.592058 6167 tablet_replica.cc:330] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: stopping tablet replica
I20260809 06:19:05.592283 6167 raft_consensus.cc:2229] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:05.592825 6167 raft_consensus.cc:2258] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:05.593948 6167 ts_tablet_manager.cc:1905] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.601974 6167 ts_tablet_manager.cc:1918] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:05.602159 6167 log.cc:1196] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/74631b175f73492ca4822f000b62d80d
I20260809 06:19:05.602581 6167 ts_tablet_manager.cc:1939] T 74631b175f73492ca4822f000b62d80d P 05351bd1156c453c80115b5ff101c8be: Deleting consensus metadata
I20260809 06:19:05.603778 5501 catalog_manager.cc:4841] TS 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979): tablet 74631b175f73492ca4822f000b62d80d (table test-table [id=2b23996034024d658528c83ac3be3ec1]) successfully deleted
Aug 09 06:19:05 dist-test-slave-2w3w krb5kdc[5468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:19:05 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:05.634268 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:05.619709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.3.137.126:44515)
0809 06:19:05.620372 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:05.620377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:05.620417 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:19:05.620603 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:05.620607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:05.620711 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:05.620914 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:05.622242 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:05.623205 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:05.624690 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:05.625103 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:05.625198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:05.629300 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:05.629305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:05.629308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:05.629341 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:05.631955 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:05.632535 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:05.632538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:05.632540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:05.632660 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:05.632983 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:05.632986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:05.632988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:05.633240 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:05.633424 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:05.633670 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 06:19:05.633848 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":327,"threads_started":1}
I20260809 06:19:05.637740 5514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:05.638409 5514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:05.644797 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d (DEFAULT_TABLE table=test-table [id=f5778992fddf4525870550eea078efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.644948 5812 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d (DEFAULT_TABLE table=test-table [id=f5778992fddf4525870550eea078efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.645359 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8ffa82225644a39d7413624f2e1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.645378 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8ffa82225644a39d7413624f2e1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.645332 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d (DEFAULT_TABLE table=test-table [id=f5778992fddf4525870550eea078efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.645666 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8ffa82225644a39d7413624f2e1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.651633 6177 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Bootstrap starting.
I20260809 06:19:05.652424 6178 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Bootstrap starting.
I20260809 06:19:05.653295 6179 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Bootstrap starting.
I20260809 06:19:05.654032 6177 tablet_bootstrap.cc:654] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.654276 6178 tablet_bootstrap.cc:654] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.655066 6179 tablet_bootstrap.cc:654] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.656405 6177 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: No bootstrap required, opened a new log
I20260809 06:19:05.656546 6177 ts_tablet_manager.cc:1397] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:05.657193 6178 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: No bootstrap required, opened a new log
I20260809 06:19:05.657328 6179 tablet_bootstrap.cc:492] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: No bootstrap required, opened a new log
I20260809 06:19:05.657354 6178 ts_tablet_manager.cc:1397] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 06:19:05.657508 6179 ts_tablet_manager.cc:1397] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:19:05.657387 6177 raft_consensus.cc:348] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.657649 6177 raft_consensus.cc:374] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.657737 6177 raft_consensus.cc:729] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.657968 6177 consensus_queue.cc:260] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.658155 6178 raft_consensus.cc:348] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.658423 6178 raft_consensus.cc:374] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.658514 6178 raft_consensus.cc:729] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.658565 6177 ts_tablet_manager.cc:1428] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 06:19:05.658428 6179 raft_consensus.cc:348] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.658694 6179 raft_consensus.cc:374] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.658795 6179 raft_consensus.cc:729] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.658743 6178 consensus_queue.cc:260] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.659116 6179 consensus_queue.cc:260] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.660084 6179 ts_tablet_manager.cc:1428] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:19:05.661653 6178 ts_tablet_manager.cc:1428] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 06:19:05.733603 6184 raft_consensus.cc:482] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:05.733819 6184 raft_consensus.cc:504] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.734669 6184 leader_election.cc:288] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:05.735263 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8ffa82225644a39d7413624f2e1b9d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9" is_pre_election: true
I20260809 06:19:05.735321 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8ffa82225644a39d7413624f2e1b9d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be" is_pre_election: true
I20260809 06:19:05.735528 5682 raft_consensus.cc:2454] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 0.
I20260809 06:19:05.735586 5987 raft_consensus.cc:2454] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 0.
I20260809 06:19:05.736131 5760 leader_election.cc:302] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05351bd1156c453c80115b5ff101c8be, 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:05.736411 6184 raft_consensus.cc:2781] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:05.736542 6184 raft_consensus.cc:482] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:05.736652 6184 raft_consensus.cc:3037] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:05.738749 6184 raft_consensus.cc:504] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.739356 6184 leader_election.cc:288] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 election: Requested vote from peers f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:05.739784 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8ffa82225644a39d7413624f2e1b9d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f9243c4978480c8986d4691b96e0d9"
I20260809 06:19:05.739928 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8ffa82225644a39d7413624f2e1b9d" candidate_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be"
I20260809 06:19:05.740020 5682 raft_consensus.cc:3037] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:05.740175 5987 raft_consensus.cc:3037] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:05.743425 5682 raft_consensus.cc:2454] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 1.
I20260809 06:19:05.743512 5987 raft_consensus.cc:2454] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed67d1bcc4b4b8ebd930e4f026f2bac in term 1.
I20260809 06:19:05.743958 5760 leader_election.cc:302] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:05.744266 6184 raft_consensus.cc:2781] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:05.744838 6184 raft_consensus.cc:686] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 LEADER]: Becoming Leader. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Running, Role: LEADER
I20260809 06:19:05.745388 6184 consensus_queue.cc:237] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.748034 5516 catalog_manager.cc:5495] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66). New cstate: current_term: 1 leader_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:05.788381 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:05.782266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.3.137.66:38519)
0809 06:19:05.782874 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:05.782878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:05.782896 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:19:05.782945 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:05.782948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:05.783081 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:05.783267 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:05.783726 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:05.784608 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:05.786537 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:05.786989 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:05.787082 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:05.787597 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:05.787707 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:05.787797 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:05.787964 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:19:05.788025 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":314,"threads_started":1}
I20260809 06:19:05.797488 5987 raft_consensus.cc:1261] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Refusing update from remote peer 4ed67d1bcc4b4b8ebd930e4f026f2bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:05.797489 5682 raft_consensus.cc:1261] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Refusing update from remote peer 4ed67d1bcc4b4b8ebd930e4f026f2bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:05.798089 6184 consensus_queue.cc:1035] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:05.798417 6186 consensus_queue.cc:1035] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:05.847991 5515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:05.848280 5515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:05.851300 5515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Sending DeleteTablet for 3 replicas of tablet 1d8ffa82225644a39d7413624f2e1b9d
I20260809 06:19:05.852068 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:39944
I20260809 06:19:05.852093 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47488
I20260809 06:19:05.852068 5812 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8ffa82225644a39d7413624f2e1b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:05 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:35622
I20260809 06:19:05.852397 6166 tablet_replica.cc:330] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: stopping tablet replica
I20260809 06:19:05.852428 6167 tablet_replica.cc:330] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: stopping tablet replica
I20260809 06:19:05.852649 6167 raft_consensus.cc:2229] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:05.852631 6166 raft_consensus.cc:2229] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:05.852851 6167 raft_consensus.cc:2258] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:05.852916 6166 raft_consensus.cc:2258] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:05.853138 6168 tablet_replica.cc:330] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: stopping tablet replica
I20260809 06:19:05.853354 6168 raft_consensus.cc:2229] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:05.853435 5515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 06:19:05.853693 6168 raft_consensus.cc:2258] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 06:19:05.854068 5515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:05.854156 6166 ts_tablet_manager.cc:1905] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.854216 6167 ts_tablet_manager.cc:1905] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.854933 6168 ts_tablet_manager.cc:1905] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:05.861629 6166 ts_tablet_manager.cc:1918] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:05.861964 6166 log.cc:1196] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/1d8ffa82225644a39d7413624f2e1b9d
I20260809 06:19:05.862362 6168 ts_tablet_manager.cc:1918] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:05.862535 6168 log.cc:1196] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/1d8ffa82225644a39d7413624f2e1b9d
I20260809 06:19:05.862706 6166 ts_tablet_manager.cc:1939] T 1d8ffa82225644a39d7413624f2e1b9d P f8f9243c4978480c8986d4691b96e0d9: Deleting consensus metadata
I20260809 06:19:05.863121 6168 ts_tablet_manager.cc:1939] T 1d8ffa82225644a39d7413624f2e1b9d P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting consensus metadata
I20260809 06:19:05.864207 6167 ts_tablet_manager.cc:1918] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:05.864406 6167 log.cc:1196] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/1d8ffa82225644a39d7413624f2e1b9d
I20260809 06:19:05.864566 5503 catalog_manager.cc:4841] TS 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519): tablet 1d8ffa82225644a39d7413624f2e1b9d (table test-table [id=f5778992fddf4525870550eea078efda]) successfully deleted
I20260809 06:19:05.864631 5812 tablet_service.cc:1467] Processing CreateTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 (DEFAULT_TABLE table=test-table [id=ea34864f9f8b40a7956a961674b79495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.864887 6167 ts_tablet_manager.cc:1939] T 1d8ffa82225644a39d7413624f2e1b9d P 05351bd1156c453c80115b5ff101c8be: Deleting consensus metadata
I20260809 06:19:05.864915 5500 catalog_manager.cc:4841] TS f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649): tablet 1d8ffa82225644a39d7413624f2e1b9d (table test-table [id=f5778992fddf4525870550eea078efda]) successfully deleted
I20260809 06:19:05.865036 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9eb6cda08d4bf49468fed2e5ef2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.865620 5965 tablet_service.cc:1467] Processing CreateTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 (DEFAULT_TABLE table=test-table [id=ea34864f9f8b40a7956a961674b79495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.865636 5659 tablet_service.cc:1467] Processing CreateTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 (DEFAULT_TABLE table=test-table [id=ea34864f9f8b40a7956a961674b79495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:05.865981 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9eb6cda08d4bf49468fed2e5ef2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.865988 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9eb6cda08d4bf49468fed2e5ef2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:05.870157 6178 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Bootstrap starting.
I20260809 06:19:05.871276 5501 catalog_manager.cc:4841] TS 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979): tablet 1d8ffa82225644a39d7413624f2e1b9d (table test-table [id=f5778992fddf4525870550eea078efda]) successfully deleted
I20260809 06:19:05.871456 6179 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Bootstrap starting.
I20260809 06:19:05.871699 6178 tablet_bootstrap.cc:654] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.873101 6177 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Bootstrap starting.
I20260809 06:19:05.873270 6179 tablet_bootstrap.cc:654] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.873888 6178 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: No bootstrap required, opened a new log
I20260809 06:19:05.874003 6178 ts_tablet_manager.cc:1397] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:19:05.874799 6177 tablet_bootstrap.cc:654] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:05.874799 6178 raft_consensus.cc:348] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.875147 6178 raft_consensus.cc:374] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.875211 6178 raft_consensus.cc:729] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.875438 6178 consensus_queue.cc:260] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.875553 6179 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: No bootstrap required, opened a new log
I20260809 06:19:05.875707 6179 ts_tablet_manager.cc:1397] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:19:05.876703 6178 ts_tablet_manager.cc:1428] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 06:19:05.877462 6179 raft_consensus.cc:348] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.878242 6177 tablet_bootstrap.cc:492] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: No bootstrap required, opened a new log
I20260809 06:19:05.878415 6177 ts_tablet_manager.cc:1397] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 06:19:05.878429 6179 raft_consensus.cc:374] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.878530 6179 raft_consensus.cc:729] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed67d1bcc4b4b8ebd930e4f026f2bac, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.878856 6179 consensus_queue.cc:260] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.879285 6177 raft_consensus.cc:348] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.879458 6179 ts_tablet_manager.cc:1428] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260809 06:19:05.879577 6177 raft_consensus.cc:374] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:05.879738 6177 raft_consensus.cc:729] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05351bd1156c453c80115b5ff101c8be, State: Initialized, Role: FOLLOWER
I20260809 06:19:05.880043 6177 consensus_queue.cc:260] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:05.881227 6177 ts_tablet_manager.cc:1428] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:06.108575 6183 raft_consensus.cc:482] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:06.108824 6183 raft_consensus.cc:504] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:06.110610 6183 leader_election.cc:288] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:06.119391 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.111143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:46829 (local address 127.3.137.66:38519)
0809 06:19:06.111377 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:06.111386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:06.112378 (+ 992us) server_negotiation.cc:407] Connection header received
0809 06:19:06.112723 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:06.112728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:06.112806 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:06.112876 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:06.113726 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.114859 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.117258 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.118473 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.118594 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.118646 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:06.119108 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 06:19:06.119206 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:19:06.119629 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.110798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:38519 (local address 127.3.137.65:46829)
0809 06:19:06.112076 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:06.112103 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:06.112539 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:06.113111 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:06.113118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:06.113153 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:06.113483 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.113497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.115041 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.115049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.117082 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.117104 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.118683 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.118690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.118859 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.118895 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:19:06.119063 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 06:19:06.119213 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":1316,"threads_started":1}
I20260809 06:19:06.120265 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9eb6cda08d4bf49468fed2e5ef2380" candidate_uuid: "f8f9243c4978480c8986d4691b96e0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" is_pre_election: true
I20260809 06:19:06.120476 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.110896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.3.137.65:48049)
0809 06:19:06.112541 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:06.112565 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:06.112898 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:06.113973 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:06.113978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:06.114002 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:06.114555 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.114568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.116240 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.116243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.118085 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.118100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.119648 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.119656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.119845 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.119881 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:19:06.119979 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 06:19:06.120130 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":741,"threads_started":1}
I20260809 06:19:06.120544 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.111384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48049 (local address 127.3.137.67:38979)
0809 06:19:06.113452 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:06.113459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:06.113485 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:19:06.113555 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:06.113559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:06.113659 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:06.113929 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:06.114745 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.116105 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.118226 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.119375 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.119552 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.119638 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:06.120013 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 06:19:06.120157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":1820,"threads_started":1}
I20260809 06:19:06.120544 5839 raft_consensus.cc:2454] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8f9243c4978480c8986d4691b96e0d9 in term 0.
I20260809 06:19:06.121187 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9eb6cda08d4bf49468fed2e5ef2380" candidate_uuid: "f8f9243c4978480c8986d4691b96e0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be" is_pre_election: true
I20260809 06:19:06.121289 5611 leader_election.cc:302] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:06.121461 5987 raft_consensus.cc:2454] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8f9243c4978480c8986d4691b96e0d9 in term 0.
I20260809 06:19:06.121752 6183 raft_consensus.cc:2781] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:06.121919 6183 raft_consensus.cc:482] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:06.122066 6183 raft_consensus.cc:3037] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:06.124706 6183 raft_consensus.cc:504] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:06.125387 6183 leader_election.cc:288] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519), 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979)
I20260809 06:19:06.125759 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9eb6cda08d4bf49468fed2e5ef2380" candidate_uuid: "f8f9243c4978480c8986d4691b96e0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac"
I20260809 06:19:06.125818 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9eb6cda08d4bf49468fed2e5ef2380" candidate_uuid: "f8f9243c4978480c8986d4691b96e0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05351bd1156c453c80115b5ff101c8be"
I20260809 06:19:06.125984 5839 raft_consensus.cc:3037] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:06.126026 5987 raft_consensus.cc:3037] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:06.128101 5839 raft_consensus.cc:2454] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8f9243c4978480c8986d4691b96e0d9 in term 1.
I20260809 06:19:06.128108 5987 raft_consensus.cc:2454] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8f9243c4978480c8986d4691b96e0d9 in term 1.
I20260809 06:19:06.128525 5611 leader_election.cc:302] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed67d1bcc4b4b8ebd930e4f026f2bac, f8f9243c4978480c8986d4691b96e0d9; no voters:
I20260809 06:19:06.128887 6183 raft_consensus.cc:2781] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:06.129366 6183 raft_consensus.cc:686] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 1 LEADER]: Becoming Leader. State: Replica: f8f9243c4978480c8986d4691b96e0d9, State: Running, Role: LEADER
I20260809 06:19:06.129738 6183 consensus_queue.cc:237] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } }
I20260809 06:19:06.133344 5516 catalog_manager.cc:5495] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "f8f9243c4978480c8986d4691b96e0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f9243c4978480c8986d4691b96e0d9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:06.187419 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.181516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.3.137.65:44649)
0809 06:19:06.182059 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:06.182064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:06.182082 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:19:06.182159 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:06.182162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:06.182236 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:06.182371 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:06.182716 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.183534 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.184963 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.185386 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.185488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.185896 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:06.186493 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:06.186601 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:06.186743 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 06:19:06.186833 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
I20260809 06:19:06.201915 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.197152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:39857 (local address 127.3.137.67:38979)
0809 06:19:06.197337 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:06.197342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:06.197666 (+ 324us) server_negotiation.cc:407] Connection header received
0809 06:19:06.197815 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:06.197819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:06.197883 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:06.197968 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:06.198425 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.199195 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.200517 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.200908 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.200999 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.201380 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:06.201473 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:06.201538 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:06.201664 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 06:19:06.201723 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 06:19:06.202061 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.196872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:38979 (local address 127.3.137.65:39857)
0809 06:19:06.197506 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:06.197521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:06.197703 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:06.197985 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:06.197988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:06.197997 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:06.198289 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.198297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.199314 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.199318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.200401 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.200413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.201056 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.201060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.201189 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.201239 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:06.201555 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:06.201569 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 06:19:06.201628 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 06:19:06.201722 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"spinlock_wait_cycles":5504,"thread_start_us":468,"threads_started":1}
I20260809 06:19:06.209873 5987 raft_consensus.cc:1261] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Refusing update from remote peer f8f9243c4978480c8986d4691b96e0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:06.210253 5839 raft_consensus.cc:1261] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Refusing update from remote peer f8f9243c4978480c8986d4691b96e0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:06.210536 6201 consensus_queue.cc:1035] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05351bd1156c453c80115b5ff101c8be" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:06.210938 6183 consensus_queue.cc:1035] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed67d1bcc4b4b8ebd930e4f026f2bac" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:06.217661 6148 mvcc.cc:205] Tried to move back new op lower bound from 7316505994069786624 to 7316505993751003136. Current Snapshot: MvccSnapshot[applied={T|T < 7316505994069786624}]
I20260809 06:19:06.237231 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.231345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:40263 (local address 127.3.137.65:44649)
0809 06:19:06.231650 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:06.231665 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:06.232338 (+ 673us) server_negotiation.cc:407] Connection header received
0809 06:19:06.232491 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:06.232494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:06.232554 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:06.232627 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:06.232992 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.233804 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.235475 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.235922 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.236077 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.236448 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:06.236547 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:06.236663 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:06.236827 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 06:19:06.236902 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:19:06.237296 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:06.231301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44649 (local address 127.3.137.67:40263)
0809 06:19:06.232120 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:06.232136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:06.232372 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:06.232630 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:06.232633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:06.232640 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:06.232860 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.232867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.233979 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.233987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.235326 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:06.235340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:06.236121 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:06.236130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:06.236251 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:06.236286 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:06.236671 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:06.236685 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 06:19:06.236788 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 06:19:06.236922 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":656,"threads_started":1}
W20260809 06:19:06.238490 6149 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 06:19:07.831333 5514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:07.831542 5514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:07.835368 5514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 697d40517f5b4d30abe863fe015aa2a1: Sending DeleteTablet for 3 replicas of tablet ff9eb6cda08d4bf49468fed2e5ef2380
I20260809 06:19:07.836125 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47488
I20260809 06:19:07.836311 5812 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:35622
I20260809 06:19:07.836524 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9eb6cda08d4bf49468fed2e5ef2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:39944
I20260809 06:19:07.836989 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5591
I20260809 06:19:07.849681 6237 tablet_replica.cc:330] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: stopping tablet replica
I20260809 06:19:07.850096 6237 raft_consensus.cc:2229] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:07.850492 6237 raft_consensus.cc:2258] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:07.851330 6236 tablet_replica.cc:330] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: stopping tablet replica
I20260809 06:19:07.851650 6236 raft_consensus.cc:2229] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:07.851866 6237 ts_tablet_manager.cc:1905] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:07.852341 6236 raft_consensus.cc:2258] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:07.853372 6236 ts_tablet_manager.cc:1905] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:07.860036 6237 ts_tablet_manager.cc:1918] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:07.860271 6237 log.cc:1196] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/ff9eb6cda08d4bf49468fed2e5ef2380
I20260809 06:19:07.860558 6236 ts_tablet_manager.cc:1918] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:07.860802 6236 log.cc:1196] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/ff9eb6cda08d4bf49468fed2e5ef2380
I20260809 06:19:07.860842 6237 ts_tablet_manager.cc:1939] T ff9eb6cda08d4bf49468fed2e5ef2380 P 4ed67d1bcc4b4b8ebd930e4f026f2bac: Deleting consensus metadata
I20260809 06:19:07.861331 6236 ts_tablet_manager.cc:1939] T ff9eb6cda08d4bf49468fed2e5ef2380 P f8f9243c4978480c8986d4691b96e0d9: Deleting consensus metadata
I20260809 06:19:07.862129 5503 catalog_manager.cc:4841] TS 4ed67d1bcc4b4b8ebd930e4f026f2bac (127.3.137.66:38519): tablet ff9eb6cda08d4bf49468fed2e5ef2380 (table test-table [id=ea34864f9f8b40a7956a961674b79495]) successfully deleted
I20260809 06:19:07.862331 6238 tablet_replica.cc:330] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: stopping tablet replica
I20260809 06:19:07.862537 6238 raft_consensus.cc:2229] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:07.862782 6238 raft_consensus.cc:2258] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:07.863131 5500 catalog_manager.cc:4841] TS f8f9243c4978480c8986d4691b96e0d9 (127.3.137.65:44649): tablet ff9eb6cda08d4bf49468fed2e5ef2380 (table test-table [id=ea34864f9f8b40a7956a961674b79495]) successfully deleted
I20260809 06:19:07.864385 6238 ts_tablet_manager.cc:1905] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:07.868778 6238 ts_tablet_manager.cc:1918] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:19:07.868885 6238 log.cc:1196] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/ff9eb6cda08d4bf49468fed2e5ef2380
I20260809 06:19:07.869202 6238 ts_tablet_manager.cc:1939] T ff9eb6cda08d4bf49468fed2e5ef2380 P 05351bd1156c453c80115b5ff101c8be: Deleting consensus metadata
I20260809 06:19:07.870138 5501 catalog_manager.cc:4841] TS 05351bd1156c453c80115b5ff101c8be (127.3.137.67:38979): tablet ff9eb6cda08d4bf49468fed2e5ef2380 (table test-table [id=ea34864f9f8b40a7956a961674b79495]) successfully deleted
Aug 09 06:19:07 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:07.878707 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:07.839616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.3.137.65:44649)
0809 06:19:07.855376 (+ 15760us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:07.855382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:07.855413 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:19:07.855476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:07.855482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:07.855558 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:07.855683 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:07.858067 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.859874 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.862434 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.863113 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.863241 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:07.873078 (+ 9837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:07.873093 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:07.873109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:07.873141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:07.876151 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.876930 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.876938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.876941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.877112 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.877468 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.877472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.877474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.877804 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:07.878042 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:07.878258 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 06:19:07.878431 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15600,"thread_start_us":15484,"threads_started":1}
W20260809 06:19:07.879429 5733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508
W20260809 06:19:07.880034 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:07.880232 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5591
I20260809 06:19:07.905076 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5742
Aug 09 06:19:07 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:07.922075 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:07.905812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.3.137.66:38519)
0809 06:19:07.906405 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:07.906409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:07.906425 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:19:07.906488 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:07.906492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:07.906550 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:07.906643 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:07.907913 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.908663 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.909956 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.910378 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.910480 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:07.914853 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:07.914870 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:07.914890 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:07.914936 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:07.918793 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.919502 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.919512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.919515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.919712 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.920265 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.920272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.920278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.920793 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:07.921134 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:07.921500 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 06:19:07.921732 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":332,"threads_started":1}
W20260809 06:19:07.922820 5882 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646
W20260809 06:19:07.923702 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5742: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:07.923852 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5742
I20260809 06:19:07.950114 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5891
Aug 09 06:19:07 dist-test-slave-2w3w krb5kdc[5468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:07.964927 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:07.950800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.3.137.67:38979)
0809 06:19:07.951411 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:07.951416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:07.951432 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:19:07.951486 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:07.951490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:07.951554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:07.951643 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:07.952883 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.953633 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.954808 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:07.955299 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:07.955402 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:07.959553 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:07.959569 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:07.959586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:07.959617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:07.962516 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.963160 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.963167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.963170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.963310 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:07.963646 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:07.963649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:07.963651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:07.964008 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:07.964215 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:07.964491 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 06:19:07.964657 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":327,"threads_started":1}
W20260809 06:19:07.965605 6029 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39976
W20260809 06:19:07.966176 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:07.966325 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5891
I20260809 06:19:07.996622 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 5482
I20260809 06:19:07.997901 5560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:08.135804 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 5482
2026-08-09T06:19:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9137 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:08 dist-test-slave-2w3w krb5kdc[6246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:08 dist-test-slave-2w3w krb5kdc[6246](info): set up 2 sockets
Aug 09 06:19:08 dist-test-slave-2w3w krb5kdc[6246](info): commencing operation
krb5kdc: starting...
W20260809 06:19:09.281075 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:09 dist-test-slave-2w3w krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:09.486723 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:40769
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:40769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:09.814409 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:09.814774 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:09.814910 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:09.822825 6259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:09.822932 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:09.823036 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:09.823112 6259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:09.823158 6259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:09.833335 6259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:40769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:40769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:09.834614 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:09.836028 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:09.848253 6267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:09.848239 6271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:09.848639 6259 server_base.cc:1034] running on GCE node
W20260809 06:19:09.848208 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:09.849609 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:09.850939 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:09.852214 6259 hybrid_clock.cc:648] HybridClock initialized: now 1786256349852164 us; error 41 us; skew 500 ppm
I20260809 06:19:09.852574 6259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:09 dist-test-slave-2w3w krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:09.855742 6259 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:09.857187 6259 webserver.cc:460] Webserver started at http://127.3.137.126:43437/ using document root <none> and password file <none>
I20260809 06:19:09.857820 6259 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:09.857919 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:09.858171 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:09.861129 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "08a9e5047038470495674c912eda4186"
format_stamp: "Formatted at 2026-08-09 06:19:09 on dist-test-slave-2w3w"
server_key: "3ed28ecd97a32f3285572a41f1fa104c"
server_key_iv: "89e5d6c362f7d332f29b84756d49de6f"
server_key_version: "encryptionkey@0"
I20260809 06:19:09.861861 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "08a9e5047038470495674c912eda4186"
format_stamp: "Formatted at 2026-08-09 06:19:09 on dist-test-slave-2w3w"
server_key: "3ed28ecd97a32f3285572a41f1fa104c"
server_key_iv: "89e5d6c362f7d332f29b84756d49de6f"
server_key_version: "encryptionkey@0"
I20260809 06:19:09.866941 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260809 06:19:09.870321 6278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:09.871709 6259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:19:09.871897 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "08a9e5047038470495674c912eda4186"
format_stamp: "Formatted at 2026-08-09 06:19:09 on dist-test-slave-2w3w"
server_key: "3ed28ecd97a32f3285572a41f1fa104c"
server_key_iv: "89e5d6c362f7d332f29b84756d49de6f"
server_key_version: "encryptionkey@0"
I20260809 06:19:09.872110 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:09.890678 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:09.894527 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:09.894832 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:09.913681 6259 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:40769
I20260809 06:19:09.913709 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:40769 every 8 connection(s)
I20260809 06:19:09.915140 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:09.916826 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6259
I20260809 06:19:09.917062 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:09.917721 3621 external_mini_cluster.cc:1428] Setting key 14f8a4e7bd890518af7d006bdbd03a66
I20260809 06:19:09.919126 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:09.932547 6334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186: Bootstrap starting.
Aug 09 06:19:09 dist-test-slave-2w3w krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:09.936457 6334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:09.937781 6334 log.cc:824] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:09.941071 6334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186: No bootstrap required, opened a new log
I20260809 06:19:09.942142 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:09.919671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.3.137.126:40769)
0809 06:19:09.920431 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:09.920445 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:09.920495 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:19:09.922486 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:09.922513 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:09.923021 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:09.923436 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:09.924725 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:09.926789 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:09.928459 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:09.929543 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:09.929740 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:09.933627 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:09.933637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:09.933654 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:09.933704 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:09.937738 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:09.938543 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:09.938549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:09.938555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:09.938723 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:09.939217 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:09.939222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:09.939225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:09.939845 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:09.940138 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:09.940449 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 06:19:09.940852 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
I20260809 06:19:09.949071 6334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } }
I20260809 06:19:09.949391 6334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:09.949471 6334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a9e5047038470495674c912eda4186, State: Initialized, Role: FOLLOWER
I20260809 06:19:09.949931 6334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } }
I20260809 06:19:09.950134 6334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:09.950225 6334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:09.950390 6334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:09.952776 6334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } }
I20260809 06:19:09.953258 6334 leader_election.cc:302] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08a9e5047038470495674c912eda4186; no voters:
I20260809 06:19:09.953884 6334 leader_election.cc:288] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:09.954020 6339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:09.954692 6339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [term 1 LEADER]: Becoming Leader. State: Replica: 08a9e5047038470495674c912eda4186, State: Running, Role: LEADER
I20260809 06:19:09.955282 6339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } }
I20260809 06:19:09.956023 6334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:09.958140 6340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08a9e5047038470495674c912eda4186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } } }
I20260809 06:19:09.958825 6340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:09.958181 6341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08a9e5047038470495674c912eda4186. Latest consensus state: current_term: 1 leader_uuid: "08a9e5047038470495674c912eda4186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a9e5047038470495674c912eda4186" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40769 } } }
I20260809 06:19:09.959604 6341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:09.965029 6348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:09.970085 6348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:09.980268 6348 catalog_manager.cc:1347] Generated new cluster ID: 37c157f9217d4895adb9ed127f9cafb9
I20260809 06:19:09.980422 6348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:09.993947 6348 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:09.995385 6348 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:10.006708 6348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08a9e5047038470495674c912eda4186: Generated new TSK 0
I20260809 06:19:10.007619 6348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:10.068066 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:10.423873 6363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:10.424142 6363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:10.424247 6363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:10.431528 6363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:10.431632 6363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:10.431744 6363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:10.441244 6363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:10.442414 6363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:10.443666 6363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:10.456393 6371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:10.456487 6368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:10.456455 6369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:10.457641 6363 server_base.cc:1034] running on GCE node
I20260809 06:19:10.458269 6363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:10.459187 6363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:10.460384 6363 hybrid_clock.cc:648] HybridClock initialized: now 1786256350460327 us; error 55 us; skew 500 ppm
I20260809 06:19:10.460702 6363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:10 dist-test-slave-2w3w krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:10.463863 6363 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:10.465379 6363 webserver.cc:460] Webserver started at http://127.3.137.65:45021/ using document root <none> and password file <none>
I20260809 06:19:10.466045 6363 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:10.466188 6363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:10.466495 6363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:10.469413 6363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "8175b622338c463aac527d5188744727"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "4fe9dca89e9521c10ab6aff8f3ec4e0b"
server_key_iv: "0249fc67ac4d38fdeee456373d4ba734"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.470152 6363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "8175b622338c463aac527d5188744727"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "4fe9dca89e9521c10ab6aff8f3ec4e0b"
server_key_iv: "0249fc67ac4d38fdeee456373d4ba734"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.475009 6363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 06:19:10.478408 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:10.479800 6363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:19:10.480000 6363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "8175b622338c463aac527d5188744727"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "4fe9dca89e9521c10ab6aff8f3ec4e0b"
server_key_iv: "0249fc67ac4d38fdeee456373d4ba734"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.480201 6363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:10.498672 6363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:10.502490 6363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:10.502784 6363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:10.503758 6363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:10.505070 6363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:10.505177 6363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:10.505332 6363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:10.505407 6363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:10.539288 6363 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:33253
I20260809 06:19:10.539336 6499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:33253 every 8 connection(s)
I20260809 06:19:10.540584 6363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:10.550808 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6363
I20260809 06:19:10.551060 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:10.551698 3621 external_mini_cluster.cc:1428] Setting key 65c3f682b4bf0beb209c85d2d9c66421
Aug 09 06:19:10 dist-test-slave-2w3w krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:10.561354 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:10.543194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48603 (local address 127.3.137.126:40769)
0809 06:19:10.543680 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:10.543685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:10.544507 (+ 822us) server_negotiation.cc:407] Connection header received
0809 06:19:10.546329 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:10.546334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:10.546423 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:10.546552 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:10.548103 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:10.548837 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:10.550385 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:10.550800 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:10.550944 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:10.554473 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:10.554478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:10.554481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:10.554512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:10.557303 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:10.558024 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:10.558036 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:10.558038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:10.558158 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:10.558603 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:10.558608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:10.558613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:10.558870 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:10.559097 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:10.560654 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0809 06:19:10.560920 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":280,"threads_started":1}
I20260809 06:19:10.561483 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:10.543638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:40769 (local address 127.3.137.65:48603)
0809 06:19:10.544334 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:10.544377 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:10.546058 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:10.546785 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:10.546793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:10.547374 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:10.547918 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:10.547935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:10.549015 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:10.549020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:10.550235 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:10.550247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:10.551070 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:10.551079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:10.551308 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:10.551976 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:10.552003 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:10.554251 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:10.557448 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:10.557465 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:10.557480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:10.557885 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:10.558272 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:10.558277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:10.558280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:10.558478 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:10.559063 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:10.559072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:10.559427 (+ 355us) client_negotiation.cc:769] Sending connection context
0809 06:19:10.559767 (+ 340us) client_negotiation.cc:240] Negotiation successful
0809 06:19:10.560343 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
I20260809 06:19:10.563138 6501 heartbeater.cc:347] Connected to a master server at 127.3.137.126:40769
I20260809 06:19:10.563444 6501 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:10.564213 6501 heartbeater.cc:511] Master 127.3.137.126:40769 requested a full tablet report, sending...
I20260809 06:19:10.566576 6296 ts_manager.cc:194] Registered new tserver with Master: 8175b622338c463aac527d5188744727 (127.3.137.65:33253)
I20260809 06:19:10.568902 6296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:48603
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:10.601883 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:10.924731 6508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:10.925040 6508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:10.925208 6508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:10.932534 6508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:10.932700 6508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:10.932894 6508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:10.942363 6508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:10.943635 6508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:10.944872 6508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:10.957834 6513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:10.957793 6514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:10.957862 6516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:10.958751 6508 server_base.cc:1034] running on GCE node
I20260809 06:19:10.959347 6508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:10.960265 6508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:10.961473 6508 hybrid_clock.cc:648] HybridClock initialized: now 1786256350961426 us; error 37 us; skew 500 ppm
I20260809 06:19:10.961777 6508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:10 dist-test-slave-2w3w krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:10.965272 6508 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:10.966749 6508 webserver.cc:460] Webserver started at http://127.3.137.66:38323/ using document root <none> and password file <none>
I20260809 06:19:10.967497 6508 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:10.967625 6508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:10.967952 6508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:10.970781 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "f66e3984a2094a5da30656470cb2392d"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "c417847be309c53df0e561f10031fbba"
server_key_iv: "88a30b09ade797574fb9f893892a9e49"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.971637 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "f66e3984a2094a5da30656470cb2392d"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "c417847be309c53df0e561f10031fbba"
server_key_iv: "88a30b09ade797574fb9f893892a9e49"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.977042 6508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:19:10.980827 6523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:10.982338 6508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:10.982511 6508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "f66e3984a2094a5da30656470cb2392d"
format_stamp: "Formatted at 2026-08-09 06:19:10 on dist-test-slave-2w3w"
server_key: "c417847be309c53df0e561f10031fbba"
server_key_iv: "88a30b09ade797574fb9f893892a9e49"
server_key_version: "encryptionkey@0"
I20260809 06:19:10.982733 6508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:11.003163 6508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:11.007709 6508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:11.008006 6508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:11.009018 6508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:11.010336 6508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:11.010412 6508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:11.010576 6508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:11.010625 6508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:11.044760 6508 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:42689
I20260809 06:19:11.044888 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:42689 every 8 connection(s)
I20260809 06:19:11.046051 6508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:11.051234 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6508
I20260809 06:19:11.051452 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:11.051997 3621 external_mini_cluster.cc:1428] Setting key ee3dae51c923ef17dacf4bdb2a1bd190
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:11.071115 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:11.049105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:36695 (local address 127.3.137.126:40769)
0809 06:19:11.049341 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:11.049348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:11.050850 (+ 1502us) server_negotiation.cc:407] Connection header received
0809 06:19:11.054923 (+ 4073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:11.054927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:11.055051 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:11.055135 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:11.057296 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.058009 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.060523 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.060977 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.061079 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:11.065330 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:11.065335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:11.065338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:11.065414 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:11.068140 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:11.068870 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:11.068875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:11.068877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:11.068998 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:11.069433 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:11.069438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:11.069440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:11.069691 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:11.069851 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:11.070519 (+ 668us) server_negotiation.cc:299] Negotiation successful
0809 06:19:11.070754 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:19:11.072242 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:11.049710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:40769 (local address 127.3.137.66:36695)
0809 06:19:11.050659 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:11.050709 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:11.054574 (+ 3865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:11.055428 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:11.055442 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:11.056288 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:11.057113 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:11.057140 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.058174 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.058182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:11.060366 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:11.060385 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.061213 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.061221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:11.061524 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:11.062313 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:11.062342 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:11.065081 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:11.068284 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:11.068294 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:11.068309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:11.068731 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:11.069114 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:11.069119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:11.069123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:11.069308 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:11.069825 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:11.069832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:11.070164 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 06:19:11.070414 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:19:11.070925 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:19:11.073742 6644 heartbeater.cc:347] Connected to a master server at 127.3.137.126:40769
I20260809 06:19:11.074005 6644 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:11.074800 6644 heartbeater.cc:511] Master 127.3.137.126:40769 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
I20260809 06:19:11.076462 6296 ts_manager.cc:194] Registered new tserver with Master: f66e3984a2094a5da30656470cb2392d (127.3.137.66:42689)
I20260809 06:19:11.077370 6296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:36695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:11.101682 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:11.417677 6649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:11.417960 6649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:11.418118 6649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:11.425346 6649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:11.425460 6649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:11.425609 6649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:11.435506 6649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:40769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:11.436782 6649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:11.438052 6649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:11.451088 6654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:11.451115 6655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:11.451156 6657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:11.452245 6649 server_base.cc:1034] running on GCE node
I20260809 06:19:11.452973 6649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:11.453890 6649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:11.455117 6649 hybrid_clock.cc:648] HybridClock initialized: now 1786256351455075 us; error 32 us; skew 500 ppm
I20260809 06:19:11.455432 6649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:11.458791 6649 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:11.460199 6649 webserver.cc:460] Webserver started at http://127.3.137.67:38205/ using document root <none> and password file <none>
I20260809 06:19:11.460825 6649 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:11.460923 6649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:11.461179 6649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:11.464025 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "a1cc2723747843159b87cd614dfad491"
format_stamp: "Formatted at 2026-08-09 06:19:11 on dist-test-slave-2w3w"
server_key: "2631f3fcc450d1a5ecde683a756ebe27"
server_key_iv: "359b26953998a10e3842f5aab3f3ff38"
server_key_version: "encryptionkey@0"
I20260809 06:19:11.464844 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "a1cc2723747843159b87cd614dfad491"
format_stamp: "Formatted at 2026-08-09 06:19:11 on dist-test-slave-2w3w"
server_key: "2631f3fcc450d1a5ecde683a756ebe27"
server_key_iv: "359b26953998a10e3842f5aab3f3ff38"
server_key_version: "encryptionkey@0"
I20260809 06:19:11.469785 6649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:19:11.473057 6665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:11.474269 6649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:11.474402 6649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "a1cc2723747843159b87cd614dfad491"
format_stamp: "Formatted at 2026-08-09 06:19:11 on dist-test-slave-2w3w"
server_key: "2631f3fcc450d1a5ecde683a756ebe27"
server_key_iv: "359b26953998a10e3842f5aab3f3ff38"
server_key_version: "encryptionkey@0"
I20260809 06:19:11.474578 6649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:11.491165 6649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:11.495573 6649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:11.495869 6649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:11.496781 6649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:11.498068 6649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:11.498176 6649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:11.498332 6649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:11.498406 6649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:11.532531 6649 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:42109
I20260809 06:19:11.532583 6784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:42109 every 8 connection(s)
I20260809 06:19:11.533998 6649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:11.540382 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6649
I20260809 06:19:11.540591 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:11.541183 3621 external_mini_cluster.cc:1428] Setting key 0c1bd9d6ee7afb8fc6f442105f44940d
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:11.555248 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:11.536733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:39587 (local address 127.3.137.126:40769)
0809 06:19:11.536952 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:11.536956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:11.538230 (+ 1274us) server_negotiation.cc:407] Connection header received
0809 06:19:11.540122 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:11.540128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:11.540216 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:11.540336 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:11.541807 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.542813 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.545348 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.545752 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.545849 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:11.549436 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:11.549442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:11.549445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:11.549480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:11.552087 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:11.552810 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:11.552817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:11.552825 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:11.552957 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:11.553404 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:11.553408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:11.553410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:11.553667 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:11.553881 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:11.554577 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 06:19:11.554848 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 06:19:11.556231 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:11.537254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:40769 (local address 127.3.137.67:39587)
0809 06:19:11.538039 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:11.538084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:11.539848 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:11.540572 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:11.540581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:11.541080 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:11.541624 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:11.541641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.543026 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.543034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:11.545191 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:11.545209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:11.545984 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:11.545990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:11.546208 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:11.546926 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:11.546950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:11.549246 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:11.552238 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:11.552246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:11.552257 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:11.552663 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:11.553071 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:11.553077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:11.553081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:11.553273 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:11.553850 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:11.553858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:11.554193 (+ 335us) client_negotiation.cc:769] Sending connection context
0809 06:19:11.554461 (+ 268us) client_negotiation.cc:240] Negotiation successful
0809 06:19:11.555011 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":306,"threads_started":1}
I20260809 06:19:11.557869 6785 heartbeater.cc:347] Connected to a master server at 127.3.137.126:40769
I20260809 06:19:11.558204 6785 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:11.559034 6785 heartbeater.cc:511] Master 127.3.137.126:40769 requested a full tablet report, sending...
I20260809 06:19:11.560426 6296 ts_manager.cc:194] Registered new tserver with Master: a1cc2723747843159b87cd614dfad491 (127.3.137.67:42109)
I20260809 06:19:11.561229 6296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:39587
I20260809 06:19:11.569502 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:19:11.571964 6501 heartbeater.cc:503] Master 127.3.137.126:40769 was elected leader, sending a full tablet report...
W20260809 06:19:11.605588 6504 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:11.603131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.3.137.126:40769)
0809 06:19:11.603394 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:11.603402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:11.603665 (+ 263us) server_negotiation.cc:407] Connection header received
0809 06:19:11.603817 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:11.603821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:11.603890 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:11.603978 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:11.605287 (+ 1309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35866: BlockingRecv error: recv got EOF from 127.0.0.1:35866 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:19:11.608466 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6363
W20260809 06:19:11.611521 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6363: Not authorized: Client connection negotiation failed: client connection to 127.3.137.65:33253: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:19:11.611706 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6363
I20260809 06:19:11.629366 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6508
W20260809 06:19:11.631994 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6508: Not authorized: Client connection negotiation failed: client connection to 127.3.137.66:42689: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:19:11.632176 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6508
I20260809 06:19:11.650863 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6649
W20260809 06:19:11.653743 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6649: Not authorized: Client connection negotiation failed: client connection to 127.3.137.67:42109: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:19:11.653884 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6649
I20260809 06:19:11.671087 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6259
I20260809 06:19:11.672272 6329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:11.773912 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6259
2026-08-09T06:19:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3636 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6803](info): set up 2 sockets
Aug 09 06:19:11 dist-test-slave-2w3w krb5kdc[6803](info): commencing operation
krb5kdc: starting...
W20260809 06:19:12.905179 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:13.111413 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39341
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:13.427369 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:13.427728 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:13.427843 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:13.434831 6818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:13.434930 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:13.435047 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:13.435114 6818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:13.435163 6818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:13.444799 6818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:13.445937 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:13.447171 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:13.458778 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:13.458748 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:13.458742 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:13.459801 6818 server_base.cc:1034] running on GCE node
I20260809 06:19:13.460508 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:13.461776 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:13.463050 6818 hybrid_clock.cc:648] HybridClock initialized: now 1786256353463005 us; error 53 us; skew 500 ppm
I20260809 06:19:13.463389 6818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:13.470716 6818 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:13.472223 6818 webserver.cc:460] Webserver started at http://127.3.137.126:34157/ using document root <none> and password file <none>
I20260809 06:19:13.472877 6818 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:13.473019 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:13.473323 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:13.476276 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "aaad4cba5626487bab38063106d79c64"
format_stamp: "Formatted at 2026-08-09 06:19:13 on dist-test-slave-2w3w"
server_key: "c751fb7989ce63204d2721f73c1a3ff1"
server_key_iv: "6096055b277b25972f3f099d49b8558a"
server_key_version: "encryptionkey@0"
I20260809 06:19:13.477047 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "aaad4cba5626487bab38063106d79c64"
format_stamp: "Formatted at 2026-08-09 06:19:13 on dist-test-slave-2w3w"
server_key: "c751fb7989ce63204d2721f73c1a3ff1"
server_key_iv: "6096055b277b25972f3f099d49b8558a"
server_key_version: "encryptionkey@0"
I20260809 06:19:13.481951 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:19:13.485178 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:13.486367 6818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 06:19:13.486548 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "aaad4cba5626487bab38063106d79c64"
format_stamp: "Formatted at 2026-08-09 06:19:13 on dist-test-slave-2w3w"
server_key: "c751fb7989ce63204d2721f73c1a3ff1"
server_key_iv: "6096055b277b25972f3f099d49b8558a"
server_key_version: "encryptionkey@0"
I20260809 06:19:13.486745 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:13.508302 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:13.512297 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:13.512589 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:13.530853 6818 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:39341
I20260809 06:19:13.530882 6887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:39341 every 8 connection(s)
I20260809 06:19:13.532203 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:13.535667 6888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:13.539690 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6818
I20260809 06:19:13.539872 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:13.540534 3621 external_mini_cluster.cc:1428] Setting key ed7bd153a3e4490a670d0bdd163015db
I20260809 06:19:13.548058 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64: Bootstrap starting.
I20260809 06:19:13.551751 6888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:13.552788 6888 log.cc:824] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:13.555575 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64: No bootstrap required, opened a new log
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:13.563520 6888 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } }
I20260809 06:19:13.563898 6888 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:13.564025 6888 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad4cba5626487bab38063106d79c64, State: Initialized, Role: FOLLOWER
I20260809 06:19:13.564342 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:13.542311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.3.137.126:39341)
0809 06:19:13.543222 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:13.543236 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:13.543289 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:19:13.546292 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:13.546320 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:13.546796 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:13.547236 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:13.548612 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:13.550661 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:13.551998 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:13.552668 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:13.552862 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:13.556976 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:13.556983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:13.557000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:13.557036 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:13.560212 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:13.560926 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:13.560937 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:13.560949 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:13.561123 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:13.561539 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:13.561544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:13.561547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:13.562077 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:13.562424 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:13.562808 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 06:19:13.563231 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":340,"threads_started":1}
I20260809 06:19:13.564692 6888 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } }
I20260809 06:19:13.564930 6888 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:13.565104 6888 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:13.565294 6888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:13.568219 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:19:13.568230 6888 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } }
I20260809 06:19:13.568753 6888 leader_election.cc:302] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaad4cba5626487bab38063106d79c64; no voters:
I20260809 06:19:13.569413 6888 leader_election.cc:288] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:13.569517 6893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:13.570815 6893 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [term 1 LEADER]: Becoming Leader. State: Replica: aaad4cba5626487bab38063106d79c64, State: Running, Role: LEADER
I20260809 06:19:13.571466 6893 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } }
I20260809 06:19:13.571708 6888 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:13.574206 6895 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaad4cba5626487bab38063106d79c64. Latest consensus state: current_term: 1 leader_uuid: "aaad4cba5626487bab38063106d79c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } } }
I20260809 06:19:13.574607 6895 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:13.574731 6894 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaad4cba5626487bab38063106d79c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad4cba5626487bab38063106d79c64" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39341 } } }
I20260809 06:19:13.575188 6894 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:13.578474 6902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:13.583600 6902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:13.593590 6902 catalog_manager.cc:1347] Generated new cluster ID: 7693ef5fdcf04d269f53b694da18f223
I20260809 06:19:13.593698 6902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:13.606160 6902 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:13.608132 6902 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:13.619709 6902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaad4cba5626487bab38063106d79c64: Generated new TSK 0
I20260809 06:19:13.620837 6902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:19:13.628610 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:13.619407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.3.137.126:39341)
0809 06:19:13.619580 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:13.619586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:13.620329 (+ 743us) server_negotiation.cc:407] Connection header received
0809 06:19:13.620517 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:13.620523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:13.620614 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:13.620728 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:13.622123 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:13.623292 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:13.624871 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:13.625543 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:13.625681 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:13.626908 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:13.626915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:13.626919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:13.627763 (+ 844us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:13.627843 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:13.627877 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:13.627991 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:13.628141 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 06:19:13.628254 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:19:13.636343 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6818
I20260809 06:19:13.637471 6883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:13.726817 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6818
2026-08-09T06:19:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1948 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6925](info): set up 2 sockets
Aug 09 06:19:13 dist-test-slave-2w3w krb5kdc[6925](info): commencing operation
krb5kdc: starting...
W20260809 06:19:14.868750 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:15 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:15.079293 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45631
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:15.401152 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:15.401437 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:15.401655 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:15.408794 6938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:15.408918 6938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:15.409024 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:15.409125 6938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:15.409211 6938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:15.418795 6938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:15.420083 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:15.421376 6938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:15.433624 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:15.433679 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:15.433934 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:15.434657 6938 server_base.cc:1034] running on GCE node
I20260809 06:19:15.435564 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:15.437086 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:15.438320 6938 hybrid_clock.cc:648] HybridClock initialized: now 1786256355438278 us; error 48 us; skew 500 ppm
I20260809 06:19:15.438673 6938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:15 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:15.441962 6938 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:15.443387 6938 webserver.cc:460] Webserver started at http://127.3.137.126:37577/ using document root <none> and password file <none>
I20260809 06:19:15.444011 6938 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:15.444113 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:15.444355 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:15.447302 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "ad269d6e60f34f45b1ad99b6e2a668f4"
format_stamp: "Formatted at 2026-08-09 06:19:15 on dist-test-slave-2w3w"
server_key: "eb47776f24f17c31ac9f35114f6e8b96"
server_key_iv: "528d73b66f70c436ce2e16d497c5c703"
server_key_version: "encryptionkey@0"
I20260809 06:19:15.448036 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "ad269d6e60f34f45b1ad99b6e2a668f4"
format_stamp: "Formatted at 2026-08-09 06:19:15 on dist-test-slave-2w3w"
server_key: "eb47776f24f17c31ac9f35114f6e8b96"
server_key_iv: "528d73b66f70c436ce2e16d497c5c703"
server_key_version: "encryptionkey@0"
I20260809 06:19:15.452912 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:19:15.456143 6955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:15.457474 6938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:15.457614 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "ad269d6e60f34f45b1ad99b6e2a668f4"
format_stamp: "Formatted at 2026-08-09 06:19:15 on dist-test-slave-2w3w"
server_key: "eb47776f24f17c31ac9f35114f6e8b96"
server_key_iv: "528d73b66f70c436ce2e16d497c5c703"
server_key_version: "encryptionkey@0"
I20260809 06:19:15.457773 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:15.471361 6938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:15.475687 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:15.475930 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:15.495383 6938 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45631
I20260809 06:19:15.495474 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45631 every 8 connection(s)
I20260809 06:19:15.496784 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:15.498241 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 6938
I20260809 06:19:15.498483 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:15.499442 3621 external_mini_cluster.cc:1428] Setting key c16d5d450edb561b86b51f3b6544a1bc
I20260809 06:19:15.501839 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:19:15 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:15.516335 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4: Bootstrap starting.
I20260809 06:19:15.520771 7010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:15.521939 7010 log.cc:824] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:15.524768 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:15.501592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.3.137.126:45631)
0809 06:19:15.502406 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:15.502417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:15.502459 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:19:15.504086 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:15.504105 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:15.504461 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:15.504772 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:15.506738 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:15.508140 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:15.510083 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:15.510713 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:15.510843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:15.515956 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:15.515968 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:15.515987 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:15.516044 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:15.520295 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:15.520967 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:15.520974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:15.520980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:15.521160 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:15.521564 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:15.521569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:15.521573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:15.522120 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:15.522465 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:15.522952 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 06:19:15.523408 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":387,"threads_started":1}
I20260809 06:19:15.525535 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4: No bootstrap required, opened a new log
I20260809 06:19:15.533592 7010 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } }
I20260809 06:19:15.533927 7010 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:15.534013 7010 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad269d6e60f34f45b1ad99b6e2a668f4, State: Initialized, Role: FOLLOWER
I20260809 06:19:15.534554 7010 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } }
I20260809 06:19:15.534771 7010 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:15.534871 7010 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:15.535072 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:15.537160 7010 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } }
I20260809 06:19:15.537601 7010 leader_election.cc:302] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad269d6e60f34f45b1ad99b6e2a668f4; no voters:
I20260809 06:19:15.538115 7010 leader_election.cc:288] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:15.538336 7015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:15.539073 7015 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [term 1 LEADER]: Becoming Leader. State: Replica: ad269d6e60f34f45b1ad99b6e2a668f4, State: Running, Role: LEADER
I20260809 06:19:15.539745 7015 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } }
I20260809 06:19:15.540241 7010 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:15.542603 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } } }
I20260809 06:19:15.543074 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:15.542860 7017 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad269d6e60f34f45b1ad99b6e2a668f4. Latest consensus state: current_term: 1 leader_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad269d6e60f34f45b1ad99b6e2a668f4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45631 } } }
I20260809 06:19:15.543220 7017 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:15.544865 7022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:15.550786 7022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:15.559326 7022 catalog_manager.cc:1347] Generated new cluster ID: 86c72e76bb4846d8979ced93998b6cd4
I20260809 06:19:15.559458 7022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:15.579342 7022 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:15.580641 7022 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:15.586839 7022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad269d6e60f34f45b1ad99b6e2a668f4: Generated new TSK 0
I20260809 06:19:15.587848 7022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:15.642889 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:15.968849 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:15.969122 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:15.969214 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:15.976339 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:15.976444 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:15.976572 7036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:15.985971 7036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:15.987192 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:15.988440 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:16.001317 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.001358 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.001391 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:16.002537 7036 server_base.cc:1034] running on GCE node
I20260809 06:19:16.003183 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:16.004047 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:16.005286 7036 hybrid_clock.cc:648] HybridClock initialized: now 1786256356005257 us; error 41 us; skew 500 ppm
I20260809 06:19:16.005592 7036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:16 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:16.008756 7036 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:16.010207 7036 webserver.cc:460] Webserver started at http://127.3.137.65:41495/ using document root <none> and password file <none>
I20260809 06:19:16.010887 7036 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:16.011059 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:16.011374 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:16.014397 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "2f23c85ccc9343b6ba2de0cb1d91f324"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "f5ef4d96c7d4d5730e68484f579e59c9"
server_key_iv: "f8b85831cc97925e92adbc9a0231202c"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.015285 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "2f23c85ccc9343b6ba2de0cb1d91f324"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "f5ef4d96c7d4d5730e68484f579e59c9"
server_key_iv: "f8b85831cc97925e92adbc9a0231202c"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.020243 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:16.023838 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.025126 7036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:16.025318 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "2f23c85ccc9343b6ba2de0cb1d91f324"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "f5ef4d96c7d4d5730e68484f579e59c9"
server_key_iv: "f8b85831cc97925e92adbc9a0231202c"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.025509 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:16.041965 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:16.046311 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:16.046617 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:16.047515 7036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:16.048820 7036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:16.048895 7036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.049067 7036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:16.049129 7036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.081343 7036 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:38275
I20260809 06:19:16.081430 7169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:38275 every 8 connection(s)
I20260809 06:19:16.082639 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:16.083962 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7036
I20260809 06:19:16.084172 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:16.084905 3621 external_mini_cluster.cc:1428] Setting key dfc567bcedfeff59244262657db473e3
Aug 09 06:19:16 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:19:16.107890 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:16.085334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:40691 (local address 127.3.137.126:45631)
0809 06:19:16.085840 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:16.085848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:16.091053 (+ 5205us) server_negotiation.cc:407] Connection header received
0809 06:19:16.092826 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:16.092834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:16.092928 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:16.093047 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:16.094492 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.095794 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.097376 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.098121 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.098258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:16.101891 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:16.101896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:16.101899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:16.101931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:16.104628 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:16.105316 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:16.105324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:16.105329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:16.105455 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:16.106063 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:16.106066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:16.106068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:16.106340 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:16.106453 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:16.107253 (+ 800us) server_negotiation.cc:299] Negotiation successful
0809 06:19:16.107441 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
I20260809 06:19:16.108758 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:16.085975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45631 (local address 127.3.137.65:40691)
0809 06:19:16.086859 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:16.086916 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:16.092550 (+ 5634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:16.093242 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:16.093250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:16.093741 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:16.094327 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:16.094344 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.095976 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.095980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:16.097220 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:16.097232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.098347 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.098352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:16.098551 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:16.099239 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:16.099261 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:16.101638 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:16.104770 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:16.104778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:16.104794 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:16.105196 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:16.105621 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:16.105626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:16.105630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:16.105930 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:16.106462 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:16.106467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:16.106829 (+ 362us) client_negotiation.cc:769] Sending connection context
0809 06:19:16.107115 (+ 286us) client_negotiation.cc:240] Negotiation successful
0809 06:19:16.107498 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":369,"threads_started":1}
I20260809 06:19:16.110651 7170 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45631
I20260809 06:19:16.110954 7170 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:16.111744 7170 heartbeater.cc:511] Master 127.3.137.126:45631 requested a full tablet report, sending...
I20260809 06:19:16.114178 6973 ts_manager.cc:194] Registered new tserver with Master: 2f23c85ccc9343b6ba2de0cb1d91f324 (127.3.137.65:38275)
I20260809 06:19:16.115957 6973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:40691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:16.135691 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:16.455408 7176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:16.455683 7176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:16.455776 7176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:16.462939 7176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:16.463060 7176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:16.463176 7176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:16.472476 7176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:16.473695 7176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:16.474946 7176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:16.487663 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.487778 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.487905 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:16.488863 7176 server_base.cc:1034] running on GCE node
I20260809 06:19:16.489519 7176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:16.490404 7176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:16.491642 7176 hybrid_clock.cc:648] HybridClock initialized: now 1786256356491601 us; error 52 us; skew 500 ppm
I20260809 06:19:16.491945 7176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:16 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:16.495229 7176 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:16.496757 7176 webserver.cc:460] Webserver started at http://127.3.137.66:41709/ using document root <none> and password file <none>
I20260809 06:19:16.497400 7176 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:16.497542 7176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:16.497852 7176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:16.500813 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "5c03f74f51264ca7b4178a81f5bca3f2"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "90ac27cdac26058a18bebeff2fab8b4a"
server_key_iv: "8df69948af156baff3e276ea653450db"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.501593 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "5c03f74f51264ca7b4178a81f5bca3f2"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "90ac27cdac26058a18bebeff2fab8b4a"
server_key_iv: "8df69948af156baff3e276ea653450db"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.506825 7176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:19:16.510282 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.511615 7176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:16.511790 7176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "5c03f74f51264ca7b4178a81f5bca3f2"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "90ac27cdac26058a18bebeff2fab8b4a"
server_key_iv: "8df69948af156baff3e276ea653450db"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.511977 7176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:16.528369 7176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:16.532881 7176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:16.533182 7176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:16.534145 7176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:16.535588 7176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:16.535662 7176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.535830 7176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:16.535882 7176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:16.569911 7176 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:37525
I20260809 06:19:16.569967 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:37525 every 8 connection(s)
I20260809 06:19:16.571270 7176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:16.575445 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7176
I20260809 06:19:16.575644 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:16.576112 3621 external_mini_cluster.cc:1428] Setting key ba860de7860c2fa0329494d50581a160
Aug 09 06:19:16 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:16.592696 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:16.573743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:51491 (local address 127.3.137.126:45631)
0809 06:19:16.573931 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:16.573935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:16.574981 (+ 1046us) server_negotiation.cc:407] Connection header received
0809 06:19:16.576956 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:16.576962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:16.577050 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:16.577175 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:16.579055 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.580304 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.581820 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.582521 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.582669 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:16.586574 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:16.586578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:16.586581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:16.586613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:16.589478 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:16.590267 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:16.590274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:16.590278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:16.590425 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:16.590887 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:16.590897 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:16.590899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:16.591233 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:16.591385 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:16.592107 (+ 722us) server_negotiation.cc:299] Negotiation successful
0809 06:19:16.592339 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:19:16.593816 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:16.574102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45631 (local address 127.3.137.66:51491)
0809 06:19:16.574798 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:16.574835 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:16.576648 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:16.577494 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:16.577505 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:16.578261 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:16.578859 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:16.578876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.580481 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.580484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:16.581671 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:16.581683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:16.582765 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:16.582771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:16.583037 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:16.583801 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:16.583830 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:16.586347 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:16.589635 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:16.589643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:16.589662 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:16.590099 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:16.590552 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:16.590556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:16.590560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:16.590760 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:16.591373 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:16.591380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:16.591743 (+ 363us) client_negotiation.cc:769] Sending connection context
0809 06:19:16.592012 (+ 269us) client_negotiation.cc:240] Negotiation successful
0809 06:19:16.592562 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":276,"threads_started":1}
I20260809 06:19:16.595412 7311 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45631
I20260809 06:19:16.595757 7311 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:16.596740 7311 heartbeater.cc:511] Master 127.3.137.126:45631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:19:16.598457 6973 ts_manager.cc:194] Registered new tserver with Master: 5c03f74f51264ca7b4178a81f5bca3f2 (127.3.137.66:37525)
I20260809 06:19:16.599361 6973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:51491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:16.626395 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:16.950932 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:16.951292 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:16.951385 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:16.958470 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:16.958572 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:16.958683 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:16.968696 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:16.969991 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:16.971331 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:16.984458 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.984428 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:16.984570 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:16.985174 7317 server_base.cc:1034] running on GCE node
I20260809 06:19:16.985716 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:16.986600 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:16.987785 7317 hybrid_clock.cc:648] HybridClock initialized: now 1786256356987732 us; error 55 us; skew 500 ppm
I20260809 06:19:16.988041 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:16 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:16.991251 7317 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:16.992599 7317 webserver.cc:460] Webserver started at http://127.3.137.67:43393/ using document root <none> and password file <none>
I20260809 06:19:16.993182 7317 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:16.993278 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:16.993530 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:16.996403 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "94f50ed3ea9b46e7abb271aa8acae677"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "c6fb1c293ec7273db5032875c802dda1"
server_key_iv: "6e32b2de2ecfc7b5b728fff7e2d3405b"
server_key_version: "encryptionkey@0"
I20260809 06:19:16.997138 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "94f50ed3ea9b46e7abb271aa8acae677"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "c6fb1c293ec7273db5032875c802dda1"
server_key_iv: "6e32b2de2ecfc7b5b728fff7e2d3405b"
server_key_version: "encryptionkey@0"
I20260809 06:19:17.002189 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:19:17.005656 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:17.007165 7317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:19:17.007311 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "94f50ed3ea9b46e7abb271aa8acae677"
format_stamp: "Formatted at 2026-08-09 06:19:16 on dist-test-slave-2w3w"
server_key: "c6fb1c293ec7273db5032875c802dda1"
server_key_iv: "6e32b2de2ecfc7b5b728fff7e2d3405b"
server_key_version: "encryptionkey@0"
I20260809 06:19:17.007503 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:17.021021 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:17.025417 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:17.025767 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:17.026744 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:17.028172 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:17.028250 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:17.028337 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:17.028385 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:17.067236 7317 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:41353
I20260809 06:19:17.067296 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:41353 every 8 connection(s)
I20260809 06:19:17.068554 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:17.076882 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7317
I20260809 06:19:17.077066 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:17.077524 3621 external_mini_cluster.cc:1428] Setting key ecd1360314ed0d179f29025fe228f78b
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256356, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:17.088934 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.071228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:37967 (local address 127.3.137.126:45631)
0809 06:19:17.071425 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:17.071429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:17.072611 (+ 1182us) server_negotiation.cc:407] Connection header received
0809 06:19:17.074493 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:17.074499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:17.074619 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:17.074748 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:17.076290 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.077157 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.079053 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.079572 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.079708 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.083628 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:17.083633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:17.083636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:17.083667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:17.086328 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.087048 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.087052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.087054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.087185 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.087554 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.087557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.087559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.087808 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:17.087919 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:17.088521 (+ 602us) server_negotiation.cc:299] Negotiation successful
0809 06:19:17.088679 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:19:17.089802 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.071692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45631 (local address 127.3.137.67:37967)
0809 06:19:17.072440 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:17.072484 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:17.074206 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:17.075051 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:17.075066 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:17.075563 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:17.076116 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:17.076133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.077313 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.077319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:17.078868 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:17.078880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.079805 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.079813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:17.080064 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.080810 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:17.080832 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:17.083398 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:17.086461 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:17.086468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:17.086477 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:17.086884 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:17.087280 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:17.087283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:17.087285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:17.087453 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:17.087909 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:17.087914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:17.088191 (+ 277us) client_negotiation.cc:769] Sending connection context
0809 06:19:17.088409 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 06:19:17.088754 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":301,"threads_started":1}
I20260809 06:19:17.091223 7455 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45631
I20260809 06:19:17.091564 7455 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:17.092332 7455 heartbeater.cc:511] Master 127.3.137.126:45631 requested a full tablet report, sending...
I20260809 06:19:17.093705 6973 ts_manager.cc:194] Registered new tserver with Master: 94f50ed3ea9b46e7abb271aa8acae677 (127.3.137.67:41353)
I20260809 06:19:17.094527 6973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:37967
I20260809 06:19:17.104768 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:19:17.119521 7170 heartbeater.cc:503] Master 127.3.137.126:45631 was elected leader, sending a full tablet report...
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256357, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256357, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:17.156653 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.142764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.3.137.126:45631)
0809 06:19:17.143057 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:17.143066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:17.143447 (+ 381us) server_negotiation.cc:407] Connection header received
0809 06:19:17.143592 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:17.143596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:17.143659 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:17.143743 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:17.145277 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.146110 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.147388 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.147782 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.147887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.151500 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:17.151505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:17.151508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:17.151539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:17.154179 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.154821 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.154825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.154827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.155036 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.155377 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.155380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.155382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.155696 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:17.155875 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:17.156136 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 06:19:17.156294 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 06:19:17.157238 6973 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38906
I20260809 06:19:17.158624 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7036
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256357, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:17.174458 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.160148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.3.137.65:38275)
0809 06:19:17.160748 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:17.160757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:17.160791 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:19:17.160846 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:17.160852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:17.161077 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:17.161285 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:17.162441 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.163522 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.164717 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.165288 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.165429 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.169197 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:17.169207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:17.169221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:17.169252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:17.172006 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.172611 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.172618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.172620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.172734 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.173061 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.173064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.173066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.173396 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:17.173611 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:17.173844 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 06:19:17.174037 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":345,"threads_started":1}
W20260809 06:19:17.175815 7165 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47528
W20260809 06:19:17.176873 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7036: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:17.177062 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7036
I20260809 06:19:17.196177 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7176
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256357, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:17.212525 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.197108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.3.137.66:37525)
0809 06:19:17.197753 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:17.197762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:17.197795 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:19:17.197866 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:17.197873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:17.198079 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:17.198231 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:17.199489 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.200535 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.202069 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.202834 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.202932 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.206917 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:17.206927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:17.206941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:17.207017 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:17.209715 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.210306 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.210313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.210315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.210439 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.210762 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.210767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.210771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.211267 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:17.211582 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:17.211901 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 06:19:17.212181 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
W20260809 06:19:17.213701 7306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53466
W20260809 06:19:17.214722 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7176: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:17.214874 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7176
I20260809 06:19:17.231496 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7317
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[6925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256357, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:17.247543 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:17.232438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.3.137.67:41353)
0809 06:19:17.233113 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:17.233119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:17.233145 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:19:17.233200 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:17.233206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:17.233397 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:17.233515 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:17.234806 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.235910 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.237116 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:17.237646 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:17.237742 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:17.242010 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:17.242020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:17.242032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:17.242062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:17.244827 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.245589 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.245596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.245598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.245749 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:17.246108 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:17.246111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:17.246113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:17.246420 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:17.246646 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:17.246897 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 06:19:17.247209 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":358,"threads_started":1}
W20260809 06:19:17.248858 7450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47638
W20260809 06:19:17.249943 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:19:17.250090 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7317
I20260809 06:19:17.268795 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 6938
I20260809 06:19:17.269932 7005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:17.372728 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 6938
2026-08-09T06:19:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3649 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[7474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[7474](info): set up 2 sockets
Aug 09 06:19:17 dist-test-slave-2w3w krb5kdc[7474](info): commencing operation
krb5kdc: starting...
W20260809 06:19:18.496393 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:18 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:18.716966 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39835
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:19.039676 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:19.039959 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:19.040135 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:19.047362 7487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:19.047461 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:19.047552 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:19.047644 7487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:19.047737 7487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:19.058671 7487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:19.059993 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:19.061224 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:19.073243 7493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:19.073220 7492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:19.073267 7495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:19.074261 7487 server_base.cc:1034] running on GCE node
I20260809 06:19:19.075110 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:19.076503 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:19.077704 7487 hybrid_clock.cc:648] HybridClock initialized: now 1786256359077650 us; error 50 us; skew 500 ppm
I20260809 06:19:19.078049 7487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:19 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:19.084663 7487 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:19.086045 7487 webserver.cc:460] Webserver started at http://127.3.137.126:41893/ using document root <none> and password file <none>
I20260809 06:19:19.086714 7487 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:19.086819 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:19.087154 7487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:19.090023 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "938bf8aba7964bceb44f32f4c78818c7"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "578f83534adb05470e7e0a8978caea4e"
server_key_iv: "af8bdcf749da59ae4299aef7f0c6a2eb"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.090752 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "938bf8aba7964bceb44f32f4c78818c7"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "578f83534adb05470e7e0a8978caea4e"
server_key_iv: "af8bdcf749da59ae4299aef7f0c6a2eb"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.095865 7487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:19:19.099129 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:19.100414 7487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:19:19.100592 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "938bf8aba7964bceb44f32f4c78818c7"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "578f83534adb05470e7e0a8978caea4e"
server_key_iv: "af8bdcf749da59ae4299aef7f0c6a2eb"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.100780 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:19.124616 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:19.128611 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:19.128908 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:19.147535 7487 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:39835
I20260809 06:19:19.147624 7555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:39835 every 8 connection(s)
I20260809 06:19:19.148969 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:19.152491 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:19.157927 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7487
I20260809 06:19:19.158176 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:19.158922 3621 external_mini_cluster.cc:1428] Setting key 7da5a97960f12f6d245420a352e0c064
I20260809 06:19:19.163070 7556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7: Bootstrap starting.
I20260809 06:19:19.166618 7556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:19.167716 7556 log.cc:824] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:19.170630 7556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7: No bootstrap required, opened a new log
Aug 09 06:19:19 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:19.181294 7556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } }
I20260809 06:19:19.181761 7556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:19.181881 7556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938bf8aba7964bceb44f32f4c78818c7, State: Initialized, Role: FOLLOWER
I20260809 06:19:19.182587 7556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } }
I20260809 06:19:19.182864 7556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:19.183071 7556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:19.183248 7556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:19.183594 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:19.161079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.3.137.126:39835)
0809 06:19:19.161912 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:19.161924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:19.161966 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:19:19.163917 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:19.163942 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:19.164340 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:19.164690 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:19.166552 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.168013 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.169652 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.170299 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.170445 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:19.175031 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:19.175042 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:19.175060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:19.175114 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:19.179168 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:19.179796 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:19.179804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:19.179814 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:19.180069 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:19.180509 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:19.180514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:19.180518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:19.181186 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:19.181501 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:19.181832 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 06:19:19.182253 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260809 06:19:19.185616 7556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } }
I20260809 06:19:19.186105 7556 leader_election.cc:302] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938bf8aba7964bceb44f32f4c78818c7; no voters:
I20260809 06:19:19.186648 7556 leader_election.cc:288] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:19.186743 7561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:19.189008 7561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [term 1 LEADER]: Becoming Leader. State: Replica: 938bf8aba7964bceb44f32f4c78818c7, State: Running, Role: LEADER
I20260809 06:19:19.189616 7556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:19.189553 7561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } }
I20260809 06:19:19.192324 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938bf8aba7964bceb44f32f4c78818c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } } }
I20260809 06:19:19.192713 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:19.193915 7568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:19.193774 7563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938bf8aba7964bceb44f32f4c78818c7. Latest consensus state: current_term: 1 leader_uuid: "938bf8aba7964bceb44f32f4c78818c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938bf8aba7964bceb44f32f4c78818c7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39835 } } }
I20260809 06:19:19.194080 7563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:19.199359 7568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:19.208078 7568 catalog_manager.cc:1347] Generated new cluster ID: fae4920769d542129a48f47faf27136a
I20260809 06:19:19.208177 7568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:19.226495 7568 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:19.227852 7568 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:19.241000 7568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938bf8aba7964bceb44f32f4c78818c7: Generated new TSK 0
I20260809 06:19:19.241847 7568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:19.298512 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:19.620054 7582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:19.620337 7582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:19.620434 7582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:19.627583 7582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:19.627696 7582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:19.627815 7582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:19.637517 7582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:19.638705 7582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:19.639979 7582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:19.652486 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:19.652561 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:19.652606 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:19.658324 7582 server_base.cc:1034] running on GCE node
I20260809 06:19:19.658900 7582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:19.659852 7582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:19.661085 7582 hybrid_clock.cc:648] HybridClock initialized: now 1786256359661024 us; error 58 us; skew 500 ppm
I20260809 06:19:19.661492 7582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:19 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:19.664737 7582 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:19.666397 7582 webserver.cc:460] Webserver started at http://127.3.137.65:35955/ using document root <none> and password file <none>
I20260809 06:19:19.667287 7582 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:19.667428 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:19.667824 7582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:19.671370 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "13a889f8bf524c04820493223d88bc93"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "e68c4d84c40399fa0f358feb9aa64488"
server_key_iv: "de443c366843ffc3f69319418e6636ce"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.672155 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "13a889f8bf524c04820493223d88bc93"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "e68c4d84c40399fa0f358feb9aa64488"
server_key_iv: "de443c366843ffc3f69319418e6636ce"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.677284 7582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:19:19.680672 7598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:19.681942 7582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:19.682111 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "13a889f8bf524c04820493223d88bc93"
format_stamp: "Formatted at 2026-08-09 06:19:19 on dist-test-slave-2w3w"
server_key: "e68c4d84c40399fa0f358feb9aa64488"
server_key_iv: "de443c366843ffc3f69319418e6636ce"
server_key_version: "encryptionkey@0"
I20260809 06:19:19.682331 7582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:19.696266 7582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:19.700594 7582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:19.700898 7582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:19.701783 7582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:19.703109 7582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:19.703207 7582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:19.703353 7582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:19.703451 7582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:19.736696 7582 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40003
I20260809 06:19:19.736742 7716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40003 every 8 connection(s)
I20260809 06:19:19.737973 7582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:19.738463 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7582
I20260809 06:19:19.738687 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:19.739570 3621 external_mini_cluster.cc:1428] Setting key cca667aeee29b3d0251fa5c1b08c6ea2
Aug 09 06:19:19 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:19.760708 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:19.740836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:47821 (local address 127.3.137.126:39835)
0809 06:19:19.741313 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:19.741321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:19.742421 (+ 1100us) server_negotiation.cc:407] Connection header received
0809 06:19:19.745337 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:19.745341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:19.745410 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:19.745502 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:19.747658 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.748358 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.750638 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.751105 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.751238 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:19.754655 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:19.754660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:19.754663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:19.754694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:19.757309 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:19.758300 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:19.758303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:19.758306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:19.758423 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:19.759035 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:19.759039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:19.759041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:19.759302 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:19.759413 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:19.760286 (+ 873us) server_negotiation.cc:299] Negotiation successful
0809 06:19:19.760451 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":338,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:19:19.762224 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:19.741340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39835 (local address 127.3.137.65:47821)
0809 06:19:19.742252 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:19.742301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:19.745038 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:19.745762 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:19.745775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:19.746629 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:19.747486 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:19.747514 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.748521 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.748528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:19.750490 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:19.750503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:19.751339 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:19.751346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:19.751599 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:19.752292 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:19.752315 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:19.754424 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:19.757489 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:19.757501 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:19.757521 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:19.758178 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:19.758553 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:19.758560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:19.758567 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:19.758856 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:19.759437 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:19.759445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:19.759890 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 06:19:19.760205 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 06:19:19.760762 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":346,"threads_started":1}
I20260809 06:19:19.763751 7717 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39835
I20260809 06:19:19.764021 7717 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
I20260809 06:19:19.764777 7717 heartbeater.cc:511] Master 127.3.137.126:39835 requested a full tablet report, sending...
I20260809 06:19:19.766855 7520 ts_manager.cc:194] Registered new tserver with Master: 13a889f8bf524c04820493223d88bc93 (127.3.137.65:40003)
I20260809 06:19:19.768287 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:47821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:19.788283 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:20.108268 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:20.108595 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:20.108776 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:20.115953 7723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:20.116092 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:20.116281 7723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:20.125739 7723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:20.127023 7723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:20.128336 7723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:20.140904 7728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:20.141008 7732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:20.141008 7730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:20.141875 7723 server_base.cc:1034] running on GCE node
I20260809 06:19:20.142405 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:20.143342 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:20.144534 7723 hybrid_clock.cc:648] HybridClock initialized: now 1786256360144490 us; error 37 us; skew 500 ppm
I20260809 06:19:20.144845 7723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:20.148289 7723 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:20.152207 7723 webserver.cc:460] Webserver started at http://127.3.137.66:46789/ using document root <none> and password file <none>
I20260809 06:19:20.152885 7723 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:20.153021 7723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:20.153327 7723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:20.156418 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "045d3a713eaa44199b9a3872ab4c72ca"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "5d17c93703dce507f519814a12a32ebf"
server_key_iv: "d8f806840709c72e4810ef157d0c9360"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.157200 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "045d3a713eaa44199b9a3872ab4c72ca"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "5d17c93703dce507f519814a12a32ebf"
server_key_iv: "d8f806840709c72e4810ef157d0c9360"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.162379 7723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260809 06:19:20.165798 7739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.167168 7723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:20.167356 7723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "045d3a713eaa44199b9a3872ab4c72ca"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "5d17c93703dce507f519814a12a32ebf"
server_key_iv: "d8f806840709c72e4810ef157d0c9360"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.167549 7723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:20.178797 7723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:20.182468 7723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:20.182785 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:20.183776 7723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:20.185040 7723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:20.185138 7723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.185308 7723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:20.185360 7723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.218943 7723 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:45937
I20260809 06:19:20.219051 7857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:45937 every 8 connection(s)
I20260809 06:19:20.220256 7723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:20.227262 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7723
I20260809 06:19:20.227460 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:20.227880 3621 external_mini_cluster.cc:1428] Setting key 773de31d29f6cf2ddf33ab6038890495
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:20.242093 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.222833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:48719 (local address 127.3.137.126:39835)
0809 06:19:20.223089 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.223094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.224215 (+ 1121us) server_negotiation.cc:407] Connection header received
0809 06:19:20.226002 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.226008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.226104 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.226239 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.227878 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.228805 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.230892 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.231358 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.231455 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.235627 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.235636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.235643 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.235690 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.238749 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.239727 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.239733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.239736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.239888 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.240368 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.240371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.240373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.240638 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.240749 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.241662 (+ 913us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.241821 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:20.243643 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.223322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39835 (local address 127.3.137.66:48719)
0809 06:19:20.224022 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:20.224059 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:20.225740 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:20.226483 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:20.226490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:20.227065 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:20.227713 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:20.227734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.228980 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.228985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:20.230729 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:20.230747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.231591 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.231602 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:20.231932 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.232655 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:20.232687 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:20.235408 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:20.238886 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:20.238897 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:20.238914 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:20.239571 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:20.239999 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:20.240007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:20.240014 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:20.240250 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:20.240769 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:20.240780 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:20.241243 (+ 463us) client_negotiation.cc:769] Sending connection context
0809 06:19:20.241570 (+ 327us) client_negotiation.cc:240] Negotiation successful
0809 06:19:20.242144 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":274,"threads_started":1}
I20260809 06:19:20.245100 7858 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39835
I20260809 06:19:20.245414 7858 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:20.246089 7858 heartbeater.cc:511] Master 127.3.137.126:39835 requested a full tablet report, sending...
I20260809 06:19:20.247717 7520 ts_manager.cc:194] Registered new tserver with Master: 045d3a713eaa44199b9a3872ab4c72ca (127.3.137.66:45937)
I20260809 06:19:20.248487 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:48719
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:20.276340 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:20.598450 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:20.598763 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:20.598932 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:20.606231 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:20.606370 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:20.606556 7863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:20.616088 7863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:20.617319 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:20.618570 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:20.631444 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:20.631861 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:20.631861 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:20.632753 7863 server_base.cc:1034] running on GCE node
I20260809 06:19:20.633467 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:20.634481 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:20.635733 7863 hybrid_clock.cc:648] HybridClock initialized: now 1786256360635684 us; error 62 us; skew 500 ppm
I20260809 06:19:20.636108 7863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:20.639251 7863 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:20.640625 7863 webserver.cc:460] Webserver started at http://127.3.137.67:41155/ using document root <none> and password file <none>
I20260809 06:19:20.641305 7863 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:20.641451 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:20.641770 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:20.644749 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "9f98391ceb934deb8c9526a1cdfeeb02"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "0a10f5f171abf186781c4379bc8d9c8a"
server_key_iv: "8dca7d0e05bfb4248ddf0db3116b7102"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.645561 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "9f98391ceb934deb8c9526a1cdfeeb02"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "0a10f5f171abf186781c4379bc8d9c8a"
server_key_iv: "8dca7d0e05bfb4248ddf0db3116b7102"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.650465 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:19:20.653957 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.655335 7863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:20.655505 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "9f98391ceb934deb8c9526a1cdfeeb02"
format_stamp: "Formatted at 2026-08-09 06:19:20 on dist-test-slave-2w3w"
server_key: "0a10f5f171abf186781c4379bc8d9c8a"
server_key_iv: "8dca7d0e05bfb4248ddf0db3116b7102"
server_key_version: "encryptionkey@0"
I20260809 06:19:20.655709 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:20.668020 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:20.673448 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:20.673746 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:20.674674 7863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:20.676239 7863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:20.676348 7863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.676501 7863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:20.676581 7863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:20.710873 7863 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:36541
I20260809 06:19:20.710919 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:36541 every 8 connection(s)
I20260809 06:19:20.712225 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:20.716187 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 7863
I20260809 06:19:20.716380 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:20.716888 3621 external_mini_cluster.cc:1428] Setting key 203adfdb5b81dbac5236695396a7b6a0
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:20.732244 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.714913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:54357 (local address 127.3.137.126:39835)
0809 06:19:20.715125 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.715131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.716329 (+ 1198us) server_negotiation.cc:407] Connection header received
0809 06:19:20.718371 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.718376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.718497 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.718620 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.720378 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.721183 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.722694 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.723160 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.723265 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.726506 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.726510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.726513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.726545 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.728996 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.729688 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.729695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.729715 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.729837 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.730264 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.730269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.730274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.730650 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.730806 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.731608 (+ 802us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.731856 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:20.733121 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.715444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39835 (local address 127.3.137.67:54357)
0809 06:19:20.716168 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:20.716209 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:20.718073 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:20.718919 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:20.718932 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:20.719506 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:20.720229 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:20.720247 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.721323 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.721327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:20.722565 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:20.722576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.723338 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.723342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:20.723537 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.724138 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:20.724159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:20.726300 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:20.729141 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:20.729150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:20.729165 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:20.729557 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:20.729950 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:20.729955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:20.729959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:20.730142 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:20.730784 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:20.730792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:20.731200 (+ 408us) client_negotiation.cc:769] Sending connection context
0809 06:19:20.731513 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 06:19:20.732046 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":317,"threads_started":1}
I20260809 06:19:20.734429 7996 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39835
I20260809 06:19:20.734689 7996 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:20.735424 7996 heartbeater.cc:511] Master 127.3.137.126:39835 requested a full tablet report, sending...
I20260809 06:19:20.736820 7520 ts_manager.cc:194] Registered new tserver with Master: 9f98391ceb934deb8c9526a1cdfeeb02 (127.3.137.67:36541)
I20260809 06:19:20.737592 7520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:54357
I20260809 06:19:20.744298 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:19:20.771066 7717 heartbeater.cc:503] Master 127.3.137.126:39835 was elected leader, sending a full tablet report...
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:20.795639 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.780523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.3.137.65:40003)
0809 06:19:20.781162 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.781170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.781195 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:19:20.781257 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.781263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.781466 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.781589 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.782956 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.784110 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.785337 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.785984 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.786072 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.790061 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.790072 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.790084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.790119 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.792887 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.793541 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.793548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.793551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.793666 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.794051 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.794057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.794060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.794386 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.794695 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.795037 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.795306 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":296,"threads_started":1}
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:20.812788 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.799903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.3.137.126:39835)
0809 06:19:20.800103 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.800107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.800124 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:20.800233 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.800237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.800294 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.800371 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.801484 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.802161 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.803402 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.803835 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.803928 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.807763 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.807768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.807771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.807801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.810141 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.810760 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.810765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.810768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.810913 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.811364 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.811369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.811372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.811679 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.811964 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.812273 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.812491 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 06:19:20.813380 7520 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36662
I20260809 06:19:20.814363 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7582
I20260809 06:19:20.826938 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.815346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.3.137.65:40003)
0809 06:19:20.815513 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.815518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.815823 (+ 305us) server_negotiation.cc:407] Connection header received
0809 06:19:20.815916 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.815919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.815975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.816053 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.817149 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.817820 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.818922 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.819421 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.819522 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.821527 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.821531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.821534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.821561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.824471 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.825173 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.825177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.825179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.825307 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.825712 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.825715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.825717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.826002 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.826188 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.826487 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.826668 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:19:20.827683 7712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:20.910403 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7582
I20260809 06:19:20.928059 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7723
Aug 09 06:19:20 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:20.943975 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:20.928960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.3.137.66:45937)
0809 06:19:20.929530 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:20.929537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:20.929563 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:19:20.929626 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:20.929633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:20.929854 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:20.929983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:20.931236 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.932341 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.933559 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:20.934164 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:20.934273 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:20.938512 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:20.938523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:20.938536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:20.938569 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:20.941326 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.941955 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.941962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.941964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.942093 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:20.942413 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:20.942416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:20.942418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:20.942787 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:20.943175 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:20.943452 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 06:19:20.943657 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":299,"threads_started":1}
I20260809 06:19:20.945219 7853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:21.028793 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7723
I20260809 06:19:21.047519 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7863
Aug 09 06:19:21 dist-test-slave-2w3w krb5kdc[7474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:21.065841 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:21.048670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.3.137.67:36541)
0809 06:19:21.049537 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:21.049547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:21.049582 (+ 35us) server_negotiation.cc:407] Connection header received
0809 06:19:21.049658 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:21.049666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:21.049982 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:21.050172 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:21.051803 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:21.053559 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:21.055041 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:21.055597 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:21.055749 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:21.060100 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:21.060112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:21.060125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:21.060155 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:21.063014 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:21.063683 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:21.063691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:21.063695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:21.063837 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:21.064239 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:21.064244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:21.064248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:21.064578 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:21.064927 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:21.065205 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 06:19:21.065465 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":450,"threads_started":1}
I20260809 06:19:21.067191 7991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:21.153808 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7863
I20260809 06:19:21.171471 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 7487
I20260809 06:19:21.172744 7550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:21.274192 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 7487
2026-08-09T06:19:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3901 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:21 dist-test-slave-2w3w krb5kdc[8016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:21 dist-test-slave-2w3w krb5kdc[8016](info): set up 2 sockets
Aug 09 06:19:21 dist-test-slave-2w3w krb5kdc[8016](info): commencing operation
krb5kdc: starting...
W20260809 06:19:22.473672 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:22 dist-test-slave-2w3w krb5kdc[8016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:22.685961 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33187
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:23.004330 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:23.004621 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:23.004801 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:23.012027 8029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:23.012149 8029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:23.012253 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:23.012363 8029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:23.012451 8029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:23.022293 8029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:23.023578 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:23.024875 8029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:23.036475 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:23.036608 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:23.036531 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:23.037685 8029 server_base.cc:1034] running on GCE node
I20260809 06:19:23.038492 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:23.039819 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:23.041023 8029 hybrid_clock.cc:648] HybridClock initialized: now 1786256363040965 us; error 47 us; skew 500 ppm
I20260809 06:19:23.041337 8029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:23 dist-test-slave-2w3w krb5kdc[8016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:23.044483 8029 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:23.045980 8029 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:23.046113 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:23.046496 8029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:23.050705 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "3e7bcbe6eadd40838a868faf96dfe536"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "152c82cbffa70bd998d61c0a31afe306"
server_key_iv: "9cd9d4625c5ee151a43aa965091bacbd"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.051878 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "3e7bcbe6eadd40838a868faf96dfe536"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "152c82cbffa70bd998d61c0a31afe306"
server_key_iv: "9cd9d4625c5ee151a43aa965091bacbd"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.057075 8029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:19:23.060524 8044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:23.061761 8029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:23.061901 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "3e7bcbe6eadd40838a868faf96dfe536"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "152c82cbffa70bd998d61c0a31afe306"
server_key_iv: "9cd9d4625c5ee151a43aa965091bacbd"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.062075 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:23.073148 8029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:23.076983 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:23.077207 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:23.096261 8029 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:33187
I20260809 06:19:23.096238 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:33187 every 8 connection(s)
I20260809 06:19:23.097131 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:23.100585 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:23.104985 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8029
I20260809 06:19:23.105223 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:23.105826 3621 external_mini_cluster.cc:1428] Setting key 3f06a8e1d58d21f3b2fc36201b85c92c
I20260809 06:19:23.111342 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: Bootstrap starting.
I20260809 06:19:23.114913 8099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:23.116170 8099 log.cc:824] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:23.119530 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: No bootstrap required, opened a new log
Aug 09 06:19:23 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:23.127583 8099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } }
I20260809 06:19:23.127967 8099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:23.128114 8099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7bcbe6eadd40838a868faf96dfe536, State: Initialized, Role: FOLLOWER
I20260809 06:19:23.128764 8099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } }
I20260809 06:19:23.129004 8099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:23.129172 8099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:23.129323 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:23.107720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.3.137.126:33187)
0809 06:19:23.108529 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:23.108543 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:23.108586 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:19:23.110302 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:23.110324 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:23.110673 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:23.111052 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:23.112931 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.114212 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.116086 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.116996 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.117191 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:23.121710 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:23.121717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:23.121730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:23.121766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:23.125092 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:23.125790 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:23.125796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:23.125801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:23.125974 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:23.126392 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:23.126396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:23.126398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:23.126948 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:23.127298 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:23.127795 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 06:19:23.128271 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":409,"threads_started":1}
I20260809 06:19:23.129366 8099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:23.131752 8099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } }
I20260809 06:19:23.132303 8099 leader_election.cc:302] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e7bcbe6eadd40838a868faf96dfe536; no voters:
I20260809 06:19:23.132856 8099 leader_election.cc:288] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:23.133001 8105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:23.134306 8105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7bcbe6eadd40838a868faf96dfe536, State: Running, Role: LEADER
I20260809 06:19:23.134814 8105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } }
I20260809 06:19:23.135057 8099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:23.137552 8107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e7bcbe6eadd40838a868faf96dfe536. Latest consensus state: current_term: 1 leader_uuid: "3e7bcbe6eadd40838a868faf96dfe536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } } }
I20260809 06:19:23.137995 8107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:23.138680 8106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e7bcbe6eadd40838a868faf96dfe536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7bcbe6eadd40838a868faf96dfe536" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33187 } } }
I20260809 06:19:23.139441 8106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:23.143476 8113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:23.148576 8113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:23.157452 8113 catalog_manager.cc:1347] Generated new cluster ID: 65bcf075e9fb43539f55925f2ceffb40
I20260809 06:19:23.157562 8113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:23.171793 8113 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:23.173053 8113 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:23.189373 8113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: Generated new TSK 0
I20260809 06:19:23.190205 8113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:23.240572 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:23.563380 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:23.563642 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:23.563817 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:23.571444 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:23.571571 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:23.571759 8128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:23.581764 8128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:23.583091 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:23.584444 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:23.600674 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:23.600723 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:23.601377 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:23.601698 8128 server_base.cc:1034] running on GCE node
I20260809 06:19:23.602464 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:23.603579 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:23.604839 8128 hybrid_clock.cc:648] HybridClock initialized: now 1786256363604758 us; error 51 us; skew 500 ppm
I20260809 06:19:23.605201 8128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:23 dist-test-slave-2w3w krb5kdc[8016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:23.608628 8128 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:23.609671 8128 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:23.609760 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:23.610010 8128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:23.612943 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "71a49c5f76004eeb99fb34ce8ef0d043"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "88859d9756d9450e05f5c4f67969ae85"
server_key_iv: "b6e1226107ebd538405d372160bb7de6"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.613773 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "71a49c5f76004eeb99fb34ce8ef0d043"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "88859d9756d9450e05f5c4f67969ae85"
server_key_iv: "b6e1226107ebd538405d372160bb7de6"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.618811 8128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:19:23.622094 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:23.623541 8128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:23.623708 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "71a49c5f76004eeb99fb34ce8ef0d043"
format_stamp: "Formatted at 2026-08-09 06:19:23 on dist-test-slave-2w3w"
server_key: "88859d9756d9450e05f5c4f67969ae85"
server_key_iv: "b6e1226107ebd538405d372160bb7de6"
server_key_version: "encryptionkey@0"
I20260809 06:19:23.623932 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:23.652992 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:23.657032 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:23.657330 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:23.658289 8128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:23.659744 8128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:23.659834 8128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:23.659914 8128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:23.659968 8128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:23.698581 8128 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:42241
I20260809 06:19:23.698637 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:42241 every 8 connection(s)
I20260809 06:19:23.699595 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:23.702116 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8128
I20260809 06:19:23.702379 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:23.703202 3621 external_mini_cluster.cc:1428] Setting key a2afb7bd7cf36f242fdfeedc534384af
Aug 09 06:19:23 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:23.721652 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:23.702135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:51717 (local address 127.3.137.126:33187)
0809 06:19:23.702655 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:23.702663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:23.703756 (+ 1093us) server_negotiation.cc:407] Connection header received
0809 06:19:23.706045 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:23.706053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:23.706136 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:23.706252 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:23.707821 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.708991 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.710501 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.711239 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.711377 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:23.715798 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:23.715803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:23.715806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:23.715838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:23.718358 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:23.719191 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:23.719196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:23.719200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:23.719326 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:23.719758 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:23.719763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:23.719766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:23.720048 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:23.720225 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:23.721010 (+ 785us) server_negotiation.cc:299] Negotiation successful
0809 06:19:23.721253 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260809 06:19:23.722566 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:23.702675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33187 (local address 127.3.137.65:51717)
0809 06:19:23.703565 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:23.703612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:23.705771 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:23.706507 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:23.706515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:23.707115 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:23.707641 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:23.707657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.709142 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.709146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:23.710338 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:23.710350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:23.711474 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:23.711479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:23.711672 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:23.712467 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:23.712500 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:23.715581 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:23.718529 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:23.718537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:23.718552 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:23.719057 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:23.719443 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:23.719448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:23.719452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:23.719635 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:23.720185 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:23.720196 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:23.720536 (+ 340us) client_negotiation.cc:769] Sending connection context
0809 06:19:23.720899 (+ 363us) client_negotiation.cc:240] Negotiation successful
0809 06:19:23.721450 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":403,"threads_started":1}
I20260809 06:19:23.724311 8265 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33187
I20260809 06:19:23.724629 8265 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:19:23.725323 8265 heartbeater.cc:511] Master 127.3.137.126:33187 requested a full tablet report, sending...
I20260809 06:19:23.727608 8062 ts_manager.cc:194] Registered new tserver with Master: 71a49c5f76004eeb99fb34ce8ef0d043 (127.3.137.65:42241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
I20260809 06:19:23.729456 8062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:51717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:23.754472 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:24.076179 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:24.076437 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:24.076532 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:24.083603 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:24.083707 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:24.083837 8271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:24.093238 8271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:24.094398 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:24.095674 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:24.108923 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:24.108536 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:24.108805 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:24.112565 8271 server_base.cc:1034] running on GCE node
I20260809 06:19:24.113229 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:24.114105 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:24.115294 8271 hybrid_clock.cc:648] HybridClock initialized: now 1786256364115262 us; error 46 us; skew 500 ppm
I20260809 06:19:24.115615 8271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:24 dist-test-slave-2w3w krb5kdc[8016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:24.123564 8271 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:24.124708 8271 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:24.124840 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:24.125154 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:24.128108 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "901840b5435946e0b3060536278d5ac9"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "1e47de9eb7787178dab7291ac61255e1"
server_key_iv: "4ec552919bb45108da1efd6e0cfe88dd"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.128979 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "901840b5435946e0b3060536278d5ac9"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "1e47de9eb7787178dab7291ac61255e1"
server_key_iv: "4ec552919bb45108da1efd6e0cfe88dd"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.133831 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 06:19:24.137154 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.138686 8271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:19:24.138887 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "901840b5435946e0b3060536278d5ac9"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "1e47de9eb7787178dab7291ac61255e1"
server_key_iv: "4ec552919bb45108da1efd6e0cfe88dd"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.139175 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:24.152992 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:24.156625 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:24.156912 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:24.157790 8271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:24.159044 8271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:24.159168 8271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.159309 8271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:24.159421 8271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.192168 8271 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41997
I20260809 06:19:24.192235 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41997 every 8 connection(s)
I20260809 06:19:24.193068 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:24.194075 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8271
I20260809 06:19:24.194285 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:24.194883 3621 external_mini_cluster.cc:1428] Setting key 346df4b49d525b52f09d0330ec387fcb
Aug 09 06:19:24 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:24.214005 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.195815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:43985 (local address 127.3.137.126:33187)
0809 06:19:24.196019 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.196024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.197320 (+ 1296us) server_negotiation.cc:407] Connection header received
0809 06:19:24.199179 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.199187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.199280 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.199393 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:24.200833 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.202004 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.203635 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.204412 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.204518 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.208364 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:24.208369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:24.208372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:24.208404 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:24.210955 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.211727 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.211732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.211738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.211875 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.212296 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.212302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.212306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.212568 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:24.212726 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.213427 (+ 701us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.213655 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:19:24.214941 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.196284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33187 (local address 127.3.137.66:43985)
0809 06:19:24.197149 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:24.197195 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:24.198860 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:24.199574 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:24.199581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:24.200119 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:24.200648 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.200665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.202165 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.202169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.203496 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.203508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.204649 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.204658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.204999 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.205749 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:24.205769 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:24.208139 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:24.211148 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:24.211157 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:24.211172 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:24.211596 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:24.211991 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:24.211996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:24.211999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:24.212172 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:24.212700 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:24.212708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:24.213049 (+ 341us) client_negotiation.cc:769] Sending connection context
0809 06:19:24.213308 (+ 259us) client_negotiation.cc:240] Negotiation successful
0809 06:19:24.213809 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":351,"threads_started":1}
I20260809 06:19:24.216459 8405 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33187
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:19:24.216813 8405 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:24.217451 8405 heartbeater.cc:511] Master 127.3.137.126:33187 requested a full tablet report, sending...
I20260809 06:19:24.219013 8062 ts_manager.cc:194] Registered new tserver with Master: 901840b5435946e0b3060536278d5ac9 (127.3.137.66:41997)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
I20260809 06:19:24.219883 8062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:43985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:24.246032 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:24.561275 8410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:24.561534 8410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:24.561628 8410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:24.568706 8410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:24.568809 8410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:24.568922 8410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:24.578575 8410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:24.580163 8410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:24.582127 8410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:24.598506 8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:24.598532 8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:24.598989 8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:24.599769 8410 server_base.cc:1034] running on GCE node
I20260809 06:19:24.600502 8410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:24.601755 8410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:24.603027 8410 hybrid_clock.cc:648] HybridClock initialized: now 1786256364602918 us; error 65 us; skew 500 ppm
I20260809 06:19:24.603368 8410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:24 dist-test-slave-2w3w krb5kdc[8016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:24.607724 8410 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:24.608911 8410 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:24.609025 8410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:24.609346 8410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:24.613003 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "f8866cbc786b426098eb71a15c351b1e"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "c41c1f272982d89353f4876a49b11388"
server_key_iv: "5a79f70b5f87a2c1a930e3cd83ddd932"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.613940 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "f8866cbc786b426098eb71a15c351b1e"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "c41c1f272982d89353f4876a49b11388"
server_key_iv: "5a79f70b5f87a2c1a930e3cd83ddd932"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.620738 8410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260809 06:19:24.625767 8424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.627283 8410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:24.627455 8410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "f8866cbc786b426098eb71a15c351b1e"
format_stamp: "Formatted at 2026-08-09 06:19:24 on dist-test-slave-2w3w"
server_key: "c41c1f272982d89353f4876a49b11388"
server_key_iv: "5a79f70b5f87a2c1a930e3cd83ddd932"
server_key_version: "encryptionkey@0"
I20260809 06:19:24.627705 8410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:24.643229 8410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:24.647218 8410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:24.647473 8410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:24.648378 8410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:24.649636 8410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:24.649758 8410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.649880 8410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:24.649983 8410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:24.682893 8410 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:39757
I20260809 06:19:24.683010 8543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:39757 every 8 connection(s)
I20260809 06:19:24.683830 8410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:24.686182 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8410
I20260809 06:19:24.686399 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:24.687018 3621 external_mini_cluster.cc:1428] Setting key ee36350d03a8f2b979dead40639b39a2
Aug 09 06:19:24 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256364, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:24.705449 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.686726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:36341 (local address 127.3.137.126:33187)
0809 06:19:24.686942 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.686948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.688605 (+ 1657us) server_negotiation.cc:407] Connection header received
0809 06:19:24.690777 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.690784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.690868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.690997 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:24.692399 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.693151 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.694643 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.695119 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.695216 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.698525 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:24.698531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:24.698536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:24.698576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:24.701998 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.703118 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.703122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.703125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.703285 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.703794 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.703797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.703799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.704069 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:24.704184 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.705044 (+ 860us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.705212 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 06:19:24.706418 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.687344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33187 (local address 127.3.137.67:36341)
0809 06:19:24.688447 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:24.688502 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:24.690483 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:24.691183 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:24.691191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:24.691696 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:24.692239 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.692258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.693284 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.693287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.694488 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.694499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.695303 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.695307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.695489 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.696085 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:24.696110 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:24.698302 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:24.702165 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:24.702174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:24.702189 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:24.703010 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:24.703418 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:24.703422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:24.703425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:24.703662 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:24.704203 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:24.704227 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:24.704627 (+ 400us) client_negotiation.cc:769] Sending connection context
0809 06:19:24.704938 (+ 311us) client_negotiation.cc:240] Negotiation successful
0809 06:19:24.705378 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":542,"threads_started":1}
I20260809 06:19:24.707810 8544 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33187
I20260809 06:19:24.708113 8544 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:24.708779 8544 heartbeater.cc:511] Master 127.3.137.126:33187 requested a full tablet report, sending...
I20260809 06:19:24.710512 8062 ts_manager.cc:194] Registered new tserver with Master: f8866cbc786b426098eb71a15c351b1e (127.3.137.67:39757)
I20260809 06:19:24.711395 8062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:36341
I20260809 06:19:24.715217 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:19:24.732450 8265 heartbeater.cc:503] Master 127.3.137.126:33187 was elected leader, sending a full tablet report...
I20260809 06:19:24.732841 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.719995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.3.137.126:33187)
0809 06:19:24.720169 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.720173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.720685 (+ 512us) server_negotiation.cc:407] Connection header received
0809 06:19:24.720893 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.720897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.720980 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.721088 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:24.722535 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.723309 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.724579 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.724994 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.725133 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.727442 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:24.727447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:24.727450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:24.727482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:24.730025 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.730676 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.730683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.730686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.730823 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:24.731272 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:24.731276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:24.731278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:24.731668 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:24.731946 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.732267 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.732493 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:19:24.736706 8061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:24.743701 8061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:24.782090 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.755642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.3.137.65:42241)
0809 06:19:24.772141 (+ 16499us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.772154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.773187 (+ 1033us) server_negotiation.cc:407] Connection header received
0809 06:19:24.773386 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.773397 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.773685 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.773898 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:24.775110 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.776875 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.779495 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.780716 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.780871 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.781145 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.781599 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.781759 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16243,"thread_start_us":16512,"threads_started":1}
I20260809 06:19:24.782132 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.755733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:42241 (local address 127.0.0.1:60814)
0809 06:19:24.773053 (+ 17320us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:24.773068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:24.773212 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:24.774173 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:24.774182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:24.774212 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:24.774923 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.774935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.777077 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.777085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.779329 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.779345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.781243 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.781248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.781368 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.781400 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:19:24.781539 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 06:19:24.781724 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17230,"mutex_wait_us":30,"thread_start_us":10253,"threads_started":1}
I20260809 06:19:24.783525 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.755114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41997 (local address 127.0.0.1:41198)
0809 06:19:24.767000 (+ 11886us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:24.767053 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:24.767328 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:24.775747 (+ 8419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:24.775766 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:24.775799 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:24.776272 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.776283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.778471 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.778477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.780702 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.780718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.782594 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.782602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.782772 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.782838 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 06:19:24.782929 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:19:24.783070 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11004,"mutex_wait_us":3,"thread_start_us":18206,"threads_started":1}
I20260809 06:19:24.783639 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.755993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:39757 (local address 127.0.0.1:41658)
0809 06:19:24.774744 (+ 18751us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:24.774784 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:24.774921 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:24.775680 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:24.775685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:24.775711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:24.776206 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.776219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.777962 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.777968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.780212 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:24.780228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.782646 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.782654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:24.782885 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.782918 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:19:24.783034 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 06:19:24.783164 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18591,"mutex_wait_us":62,"thread_start_us":18401,"threads_started":1}
I20260809 06:19:24.783537 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.755347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.3.137.66:41997)
0809 06:19:24.775217 (+ 19870us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.775225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.775254 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:19:24.775339 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.775345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.775521 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.776503 (+ 982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:24.776732 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.778310 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.780894 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.782350 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.782477 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.782669 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.783046 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.783198 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19676,"thread_start_us":19514,"threads_started":1}
I20260809 06:19:24.785063 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:24.756685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.3.137.67:39757)
0809 06:19:24.774217 (+ 17532us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:24.774225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:24.775043 (+ 818us) server_negotiation.cc:407] Connection header received
0809 06:19:24.775242 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:24.775249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:24.775467 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:24.775676 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:24.776440 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.777832 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.780407 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:24.781839 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:24.781962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:24.782805 (+ 843us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:24.783983 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0809 06:19:24.784701 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17364,"thread_start_us":17185,"threads_started":1}
I20260809 06:19:24.786481 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 (DEFAULT_TABLE table=test-table [id=79284df76cbf44bfa4dd7918f4712b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:24.787024 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 (DEFAULT_TABLE table=test-table [id=79284df76cbf44bfa4dd7918f4712b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:24.788110 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ef5d533fb44e9aa190d31380e1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:24.788141 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ef5d533fb44e9aa190d31380e1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:24.788499 8474 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 (DEFAULT_TABLE table=test-table [id=79284df76cbf44bfa4dd7918f4712b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:24.790143 8474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ef5d533fb44e9aa190d31380e1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:24.799539 8562 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Bootstrap starting.
I20260809 06:19:24.802549 8563 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Bootstrap starting.
I20260809 06:19:24.803397 8562 tablet_bootstrap.cc:654] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:24.804652 8564 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Bootstrap starting.
I20260809 06:19:24.804859 8562 log.cc:824] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:24.806423 8563 tablet_bootstrap.cc:654] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:24.807685 8563 log.cc:824] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:24.807818 8564 tablet_bootstrap.cc:654] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:24.808212 8562 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: No bootstrap required, opened a new log
I20260809 06:19:24.808509 8562 ts_tablet_manager.cc:1397] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260809 06:19:24.809011 8564 log.cc:824] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:24.810683 8563 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: No bootstrap required, opened a new log
I20260809 06:19:24.810935 8563 ts_tablet_manager.cc:1397] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:19:24.811615 8564 tablet_bootstrap.cc:492] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: No bootstrap required, opened a new log
I20260809 06:19:24.811863 8564 ts_tablet_manager.cc:1397] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 06:19:24.819052 8564 raft_consensus.cc:348] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.818984 8562 raft_consensus.cc:348] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.819384 8564 raft_consensus.cc:374] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:24.819388 8562 raft_consensus.cc:374] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:24.819475 8562 raft_consensus.cc:729] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901840b5435946e0b3060536278d5ac9, State: Initialized, Role: FOLLOWER
I20260809 06:19:24.819478 8564 raft_consensus.cc:729] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8866cbc786b426098eb71a15c351b1e, State: Initialized, Role: FOLLOWER
I20260809 06:19:24.820075 8564 consensus_queue.cc:260] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.820097 8562 consensus_queue.cc:260] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.821166 8544 heartbeater.cc:503] Master 127.3.137.126:33187 was elected leader, sending a full tablet report...
I20260809 06:19:24.821803 8564 ts_tablet_manager.cc:1428] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 06:19:24.822290 8563 raft_consensus.cc:348] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.822804 8563 raft_consensus.cc:374] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:24.823616 8405 heartbeater.cc:503] Master 127.3.137.126:33187 was elected leader, sending a full tablet report...
I20260809 06:19:24.823652 8563 raft_consensus.cc:729] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a49c5f76004eeb99fb34ce8ef0d043, State: Initialized, Role: FOLLOWER
I20260809 06:19:24.824242 8562 ts_tablet_manager.cc:1428] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 06:19:24.824388 8563 consensus_queue.cc:260] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:24.826324 8563 ts_tablet_manager.cc:1428] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260809 06:19:24.935799 8545 tablet.cc:2369] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:19:24.945003 8406 tablet.cc:2369] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:19:24.951800 8266 tablet.cc:2369] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:25.166445 8570 raft_consensus.cc:482] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:25.166787 8570 raft_consensus.cc:504] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:25.168567 8570 leader_election.cc:288] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 901840b5435946e0b3060536278d5ac9 (127.3.137.66:41997), f8866cbc786b426098eb71a15c351b1e (127.3.137.67:39757)
I20260809 06:19:25.175529 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.169075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:55525 (local address 127.3.137.67:39757)
0809 06:19:25.169305 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.169313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.170061 (+ 748us) server_negotiation.cc:407] Connection header received
0809 06:19:25.170303 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.170310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.170377 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.170479 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:25.171138 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.171881 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.173758 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.174524 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.174735 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.174817 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.175143 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.175272 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:25.175613 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.169026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:60109 (local address 127.3.137.66:41997)
0809 06:19:25.169288 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.169297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.170044 (+ 747us) server_negotiation.cc:407] Connection header received
0809 06:19:25.170264 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.170268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.170330 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.170434 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:25.171072 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.171983 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.173817 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.174576 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.174772 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.174862 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.175220 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.175346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 06:19:25.175714 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.168784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41997 (local address 127.3.137.65:60109)
0809 06:19:25.169910 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:25.169929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:25.170102 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:25.170450 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:25.170455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:25.170478 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:25.170884 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:25.170893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.172116 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.172122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:25.173654 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:25.173666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.174841 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.174847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:25.175062 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.175085 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:19:25.175189 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 06:19:25.175319 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":20,"thread_start_us":739,"threads_started":1}
I20260809 06:19:25.176090 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.169055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:39757 (local address 127.3.137.65:55525)
0809 06:19:25.169934 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:25.169955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:25.170156 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:25.170504 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:25.170508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:25.170539 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:25.170935 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:25.170944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.172031 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.172038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:25.173594 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:25.173607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.174777 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.174784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:25.174978 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.175003 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:19:25.175115 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 06:19:25.175236 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":35,"spinlock_wait_cycles":23936,"thread_start_us":562,"threads_started":1}
I20260809 06:19:25.176719 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ef5d533fb44e9aa190d31380e1c87" candidate_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901840b5435946e0b3060536278d5ac9" is_pre_election: true
I20260809 06:19:25.176976 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ef5d533fb44e9aa190d31380e1c87" candidate_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8866cbc786b426098eb71a15c351b1e" is_pre_election: true
I20260809 06:19:25.177232 8357 raft_consensus.cc:2454] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a49c5f76004eeb99fb34ce8ef0d043 in term 0.
I20260809 06:19:25.177454 8497 raft_consensus.cc:2454] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a49c5f76004eeb99fb34ce8ef0d043 in term 0.
I20260809 06:19:25.177994 8148 leader_election.cc:302] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a49c5f76004eeb99fb34ce8ef0d043, 901840b5435946e0b3060536278d5ac9; no voters:
I20260809 06:19:25.178404 8570 raft_consensus.cc:2781] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:25.178555 8570 raft_consensus.cc:482] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:25.178668 8570 raft_consensus.cc:3037] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:25.180912 8570 raft_consensus.cc:504] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:25.181524 8570 leader_election.cc:288] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [CANDIDATE]: Term 1 election: Requested vote from peers 901840b5435946e0b3060536278d5ac9 (127.3.137.66:41997), f8866cbc786b426098eb71a15c351b1e (127.3.137.67:39757)
I20260809 06:19:25.181962 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ef5d533fb44e9aa190d31380e1c87" candidate_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901840b5435946e0b3060536278d5ac9"
I20260809 06:19:25.182010 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ef5d533fb44e9aa190d31380e1c87" candidate_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8866cbc786b426098eb71a15c351b1e"
I20260809 06:19:25.182194 8357 raft_consensus.cc:3037] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:25.182193 8497 raft_consensus.cc:3037] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:25.185444 8357 raft_consensus.cc:2454] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a49c5f76004eeb99fb34ce8ef0d043 in term 1.
I20260809 06:19:25.185444 8497 raft_consensus.cc:2454] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a49c5f76004eeb99fb34ce8ef0d043 in term 1.
I20260809 06:19:25.185899 8147 leader_election.cc:302] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a49c5f76004eeb99fb34ce8ef0d043, f8866cbc786b426098eb71a15c351b1e; no voters:
I20260809 06:19:25.186247 8570 raft_consensus.cc:2781] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:25.186685 8570 raft_consensus.cc:686] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 1 LEADER]: Becoming Leader. State: Replica: 71a49c5f76004eeb99fb34ce8ef0d043, State: Running, Role: LEADER
I20260809 06:19:25.187307 8570 consensus_queue.cc:237] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } }
I20260809 06:19:25.192098 8061 catalog_manager.cc:5495] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a49c5f76004eeb99fb34ce8ef0d043 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a49c5f76004eeb99fb34ce8ef0d043" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 42241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:25.238252 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.232510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.3.137.65:42241)
0809 06:19:25.232731 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.232737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.232848 (+ 111us) server_negotiation.cc:407] Connection header received
0809 06:19:25.233124 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.233128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.233215 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.233372 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:25.233785 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.234517 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.236027 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.236536 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.236675 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.237165 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:25.237655 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:25.237816 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.237898 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.237989 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:19:25.246511 8497 raft_consensus.cc:1261] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 1 FOLLOWER]: Refusing update from remote peer 71a49c5f76004eeb99fb34ce8ef0d043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:25.246490 8357 raft_consensus.cc:1261] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 1 FOLLOWER]: Refusing update from remote peer 71a49c5f76004eeb99fb34ce8ef0d043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:25.247485 8574 consensus_queue.cc:1035] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8866cbc786b426098eb71a15c351b1e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:25.247875 8570 consensus_queue.cc:1035] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901840b5435946e0b3060536278d5ac9" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:25.275326 8062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:25.275543 8062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:25.278861 8062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e7bcbe6eadd40838a868faf96dfe536: Sending DeleteTablet for 3 replicas of tablet 8e1ef5d533fb44e9aa190d31380e1c87
I20260809 06:19:25.280030 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:41198
I20260809 06:19:25.280042 8474 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:41658
I20260809 06:19:25.280751 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1ef5d533fb44e9aa190d31380e1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:60814
I20260809 06:19:25.280896 8587 tablet_replica.cc:330] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: stopping tablet replica
I20260809 06:19:25.281318 8587 raft_consensus.cc:2229] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:25.281801 8587 raft_consensus.cc:2258] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:25.282706 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8128
I20260809 06:19:25.288100 8587 ts_tablet_manager.cc:1905] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:25.291994 8589 tablet_replica.cc:330] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: stopping tablet replica
I20260809 06:19:25.292534 8589 raft_consensus.cc:2229] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:25.293411 8589 raft_consensus.cc:2258] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:25.294384 8588 tablet_replica.cc:330] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: stopping tablet replica
I20260809 06:19:25.294811 8588 raft_consensus.cc:2229] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:25.295316 8588 raft_consensus.cc:2258] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:25.297683 8589 ts_tablet_manager.cc:1905] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:25.299183 8587 ts_tablet_manager.cc:1918] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:25.299407 8587 log.cc:1196] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/8e1ef5d533fb44e9aa190d31380e1c87
I20260809 06:19:25.300052 8587 ts_tablet_manager.cc:1939] T 8e1ef5d533fb44e9aa190d31380e1c87 P 901840b5435946e0b3060536278d5ac9: Deleting consensus metadata
I20260809 06:19:25.301908 8049 catalog_manager.cc:4841] TS 901840b5435946e0b3060536278d5ac9 (127.3.137.66:41997): tablet 8e1ef5d533fb44e9aa190d31380e1c87 (table test-table [id=79284df76cbf44bfa4dd7918f4712b21]) successfully deleted
I20260809 06:19:25.303864 8588 ts_tablet_manager.cc:1905] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:25.305701 8589 ts_tablet_manager.cc:1918] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:25.305891 8589 log.cc:1196] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/8e1ef5d533fb44e9aa190d31380e1c87
Aug 09 06:19:25 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:25.306478 8589 ts_tablet_manager.cc:1939] T 8e1ef5d533fb44e9aa190d31380e1c87 P 71a49c5f76004eeb99fb34ce8ef0d043: Deleting consensus metadata
I20260809 06:19:25.308482 8045 catalog_manager.cc:4841] TS 71a49c5f76004eeb99fb34ce8ef0d043 (127.3.137.65:42241): tablet 8e1ef5d533fb44e9aa190d31380e1c87 (table test-table [id=79284df76cbf44bfa4dd7918f4712b21]) successfully deleted
I20260809 06:19:25.311237 8588 ts_tablet_manager.cc:1918] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:25.311408 8588 log.cc:1196] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/8e1ef5d533fb44e9aa190d31380e1c87
I20260809 06:19:25.311926 8588 ts_tablet_manager.cc:1939] T 8e1ef5d533fb44e9aa190d31380e1c87 P f8866cbc786b426098eb71a15c351b1e: Deleting consensus metadata
I20260809 06:19:25.313642 8048 catalog_manager.cc:4841] TS f8866cbc786b426098eb71a15c351b1e (127.3.137.67:39757): tablet 8e1ef5d533fb44e9aa190d31380e1c87 (table test-table [id=79284df76cbf44bfa4dd7918f4712b21]) successfully deleted
I20260809 06:19:25.314050 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.285828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.3.137.65:42241)
0809 06:19:25.286040 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.286050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.294673 (+ 8623us) server_negotiation.cc:407] Connection header received
0809 06:19:25.295164 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.295169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.295293 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.295376 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:25.297293 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.298426 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.300309 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.300984 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.301077 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.307683 (+ 6606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:25.307696 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:25.307714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:25.307754 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:25.311041 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.311821 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.311831 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.311834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.311983 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.312393 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.312405 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.312408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.312740 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:25.313091 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.313390 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.313660 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:19:25.314831 8260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:25.412815 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8128
I20260809 06:19:25.432825 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8271
Aug 09 06:19:25 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:25.448997 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.433621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.3.137.66:41997)
0809 06:19:25.433874 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.433881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.433902 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:25.434110 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.434116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.434203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.434319 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:25.436010 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.436731 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.438072 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.438476 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.438615 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.442856 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:25.442870 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:25.442886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:25.442918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:25.445810 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.446635 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.446643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.446645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.446797 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.447375 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.447383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.447387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.447773 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:25.448082 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.448404 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.448631 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:25.449684 8400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:25.539954 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8271
I20260809 06:19:25.560591 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8410
Aug 09 06:19:25 dist-test-slave-2w3w krb5kdc[8016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:25.575814 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:25.561417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.3.137.67:39757)
0809 06:19:25.561601 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:25.561606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:25.561624 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:19:25.561828 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:25.561833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:25.561907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:25.562001 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:25.563559 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.564272 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.565614 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:25.566020 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:25.566131 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:25.570386 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:25.570400 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:25.570416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:25.570463 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:25.573320 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.573878 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.573885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.573887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.574015 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:25.574346 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:25.574349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:25.574351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:25.574696 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:25.574926 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:25.575249 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 06:19:25.575405 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:19:25.576490 8539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:25.664947 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8410
I20260809 06:19:25.682040 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8029
I20260809 06:19:25.683300 8094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:25.793802 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8029
2026-08-09T06:19:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4520 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T06:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:25Z Disabled control of system clock
I20260809 06:19:25.865413 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43013
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 06:19:26.190201 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:26.190474 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:26.190559 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:26.197830 8598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:26.197933 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:26.197973 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:26.198031 8598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:26.198076 8598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:26.207832 8598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:26.209039 8598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:26.210765 8598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:26.222383 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:26.222476 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:26.222486 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:26.223562 8598 server_base.cc:1034] running on GCE node
I20260809 06:19:26.224339 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:26.225641 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:26.226852 8598 hybrid_clock.cc:648] HybridClock initialized: now 1786256366226803 us; error 39 us; skew 500 ppm
I20260809 06:19:26.227218 8598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:26.229437 8598 webserver.cc:460] Webserver started at http://127.3.137.126:45433/ using document root <none> and password file <none>
I20260809 06:19:26.230118 8598 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:26.230235 8598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:26.230592 8598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:26.233456 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "3021cf8db44847b5a3f4ad4b3b689fc8"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "fc3dd38a57a7e1e7e34ecca06f4208b5"
server_key_iv: "6c375b880acd6f0370d92b706e2e81df"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.234270 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "3021cf8db44847b5a3f4ad4b3b689fc8"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "fc3dd38a57a7e1e7e34ecca06f4208b5"
server_key_iv: "6c375b880acd6f0370d92b706e2e81df"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.239291 8598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 06:19:26.242524 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:26.243824 8598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:26.244010 8598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "3021cf8db44847b5a3f4ad4b3b689fc8"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "fc3dd38a57a7e1e7e34ecca06f4208b5"
server_key_iv: "6c375b880acd6f0370d92b706e2e81df"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.244225 8598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:26.249715 8598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:26.250438 8598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:26.250734 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:26.268510 8598 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:43013
I20260809 06:19:26.268555 8669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:43013 every 8 connection(s)
I20260809 06:19:26.269728 8598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:26.273222 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:26.274026 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8598
I20260809 06:19:26.274238 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:26.275139 3621 external_mini_cluster.cc:1428] Setting key d617f9a07d8dcbcdc964e68a4568229f
I20260809 06:19:26.283388 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:26.277238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.3.137.126:43013)
0809 06:19:26.278212 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:26.278225 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:26.278263 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:19:26.279905 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:26.279932 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:26.280360 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:26.280743 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:26.281791 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:26.281796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:26.281808 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:26.281828 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:26.281878 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:26.281893 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:26.282122 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:26.282254 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 06:19:26.282493 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260809 06:19:26.286881 8670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: Bootstrap starting.
I20260809 06:19:26.290092 8670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:26.291147 8670 log.cc:824] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:26.293706 8670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: No bootstrap required, opened a new log
I20260809 06:19:26.301159 8670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } }
I20260809 06:19:26.301591 8670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:26.301714 8670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3021cf8db44847b5a3f4ad4b3b689fc8, State: Initialized, Role: FOLLOWER
I20260809 06:19:26.302309 8670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } }
I20260809 06:19:26.302567 8670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:26.302706 8670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:26.302882 8670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:26.304972 8670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } }
I20260809 06:19:26.305464 8670 leader_election.cc:302] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3021cf8db44847b5a3f4ad4b3b689fc8; no voters:
I20260809 06:19:26.306031 8670 leader_election.cc:288] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:26.306166 8675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:26.306905 8675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 3021cf8db44847b5a3f4ad4b3b689fc8, State: Running, Role: LEADER
I20260809 06:19:26.307430 8675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } }
I20260809 06:19:26.308086 8670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:26.310369 8676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } } }
I20260809 06:19:26.310889 8676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:26.311290 8677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3021cf8db44847b5a3f4ad4b3b689fc8. Latest consensus state: current_term: 1 leader_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3021cf8db44847b5a3f4ad4b3b689fc8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43013 } } }
I20260809 06:19:26.311694 8677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:26.319841 8681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:26.324990 8681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:26.332419 8681 catalog_manager.cc:1347] Generated new cluster ID: 1f3d299d08474fefa02388cccec09673
I20260809 06:19:26.332535 8681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:26.356169 8681 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:26.356600 8681 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:26.361250 8681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: Generated new TSK 0
I20260809 06:19:26.361966 8681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:19:26.371749 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--builtin_ntp_servers=127.3.137.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 06:19:26.701033 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:26.701318 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:26.701411 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:26.708937 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:26.709043 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:26.709203 8695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:26.718681 8695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:26.719981 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:26.721691 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:26.735176 8704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:26.735248 8702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:26.735273 8701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:26.736210 8695 server_base.cc:1034] running on GCE node
I20260809 06:19:26.736908 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:26.737911 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:26.739065 8695 hybrid_clock.cc:648] HybridClock initialized: now 1786256366739018 us; error 38 us; skew 500 ppm
I20260809 06:19:26.739415 8695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:26.741668 8695 webserver.cc:460] Webserver started at http://127.3.137.65:38851/ using document root <none> and password file <none>
I20260809 06:19:26.742396 8695 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:26.742519 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:26.742862 8695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:26.745739 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "2ff24705ed0c4b8298f446ad26e15411"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "531a1291ba6c8a7781fe1d160f826401"
server_key_iv: "30d9f18ce514cfeea5944e47b4ecffe8"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.746551 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "2ff24705ed0c4b8298f446ad26e15411"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "531a1291ba6c8a7781fe1d160f826401"
server_key_iv: "30d9f18ce514cfeea5944e47b4ecffe8"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.751545 8695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:19:26.754885 8711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:26.756255 8695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:26.756465 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "2ff24705ed0c4b8298f446ad26e15411"
format_stamp: "Formatted at 2026-08-09 06:19:26 on dist-test-slave-2w3w"
server_key: "531a1291ba6c8a7781fe1d160f826401"
server_key_iv: "30d9f18ce514cfeea5944e47b4ecffe8"
server_key_version: "encryptionkey@0"
I20260809 06:19:26.756680 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:26.761987 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:26.762691 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:26.762951 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:26.763900 8695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:26.765319 8695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:26.765445 8695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:26.765568 8695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:26.765653 8695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:26.798313 8695 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:45143
I20260809 06:19:26.798363 8830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:45143 every 8 connection(s)
I20260809 06:19:26.799667 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:26.804023 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8695
I20260809 06:19:26.804219 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:26.805068 3621 external_mini_cluster.cc:1428] Setting key 793038bb9046a05dabd4373c25a84e2b
I20260809 06:19:26.808148 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--builtin_ntp_servers=127.3.137.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 06:19:26.808651 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:26.802140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:59727 (local address 127.3.137.126:43013)
0809 06:19:26.802611 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:26.802615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:26.803708 (+ 1093us) server_negotiation.cc:407] Connection header received
0809 06:19:26.806114 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:26.806118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:26.806215 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:26.806313 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:26.807761 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:26.807767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:26.807805 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:26.807842 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:26.807894 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:26.807910 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:26.808038 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:26.808231 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 06:19:26.808340 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
I20260809 06:19:26.809623 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:26.802549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43013 (local address 127.3.137.65:59727)
0809 06:19:26.803447 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:26.803552 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:26.805809 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:26.806575 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:26.806588 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:26.806700 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:26.807486 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:26.807488 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:26.807524 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:26.807601 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:26.808015 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:26.808027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:26.808034 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:19:26.808313 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 06:19:26.808558 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":265,"threads_started":1}
I20260809 06:19:26.811100 8831 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43013
I20260809 06:19:26.811385 8831 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:26.811928 8831 heartbeater.cc:511] Master 127.3.137.126:43013 requested a full tablet report, sending...
I20260809 06:19:26.813927 8632 ts_manager.cc:194] Registered new tserver with Master: 2ff24705ed0c4b8298f446ad26e15411 (127.3.137.65:45143)
W20260809 06:19:27.135510 8835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:27.135778 8835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:27.135869 8835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:27.142839 8835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:27.142930 8835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:27.143098 8835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:27.152024 8835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:27.153174 8835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:27.154879 8835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:27.167937 8845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:27.167905 8843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:27.167912 8842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:27.168937 8835 server_base.cc:1034] running on GCE node
I20260809 06:19:27.169487 8835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:27.170404 8835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:27.171625 8835 hybrid_clock.cc:648] HybridClock initialized: now 1786256367171575 us; error 85 us; skew 500 ppm
I20260809 06:19:27.171924 8835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:27.174175 8835 webserver.cc:460] Webserver started at http://127.3.137.66:41973/ using document root <none> and password file <none>
I20260809 06:19:27.174840 8835 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:27.174998 8835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:27.175315 8835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:27.178017 8835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "52f0062bdc704ce78deeb5697f3e32f0"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "f26540145db3121c4e293002f77477a1"
server_key_iv: "f64a218d7b4bf0a9df50371593c0cfdf"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.178813 8835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "52f0062bdc704ce78deeb5697f3e32f0"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "f26540145db3121c4e293002f77477a1"
server_key_iv: "f64a218d7b4bf0a9df50371593c0cfdf"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.183969 8835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 06:19:27.187312 8852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.188520 8835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:19:27.188694 8835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "52f0062bdc704ce78deeb5697f3e32f0"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "f26540145db3121c4e293002f77477a1"
server_key_iv: "f64a218d7b4bf0a9df50371593c0cfdf"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.188932 8835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:27.194442 8835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:27.195292 8835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:27.195564 8835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:27.196440 8835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:27.197743 8835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:27.197868 8835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.198004 8835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:27.198096 8835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.231995 8835 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40665
I20260809 06:19:27.232043 8972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40665 every 8 connection(s)
I20260809 06:19:27.233335 8835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:27.239588 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8835
I20260809 06:19:27.239751 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:27.240360 3621 external_mini_cluster.cc:1428] Setting key d84f6a3e7799383664031a28dd5e5d8b
I20260809 06:19:27.241577 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.235964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:47071 (local address 127.3.137.126:43013)
0809 06:19:27.236137 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.236142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.237434 (+ 1292us) server_negotiation.cc:407] Connection header received
0809 06:19:27.239149 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.239154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.239207 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.239290 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.240759 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.240780 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.240784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.240802 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.240834 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.240846 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.240967 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.241191 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.241266 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:19:27.242550 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.236379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43013 (local address 127.3.137.66:47071)
0809 06:19:27.237240 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.237285 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.238814 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.239577 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.239590 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.239709 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.240426 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.240430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.240479 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.240585 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.240955 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.240967 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.240974 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.241212 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.241476 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260809 06:19:27.243615 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--builtin_ntp_servers=127.3.137.84:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 06:19:27.244001 8973 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43013
I20260809 06:19:27.244302 8973 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:27.244838 8973 heartbeater.cc:511] Master 127.3.137.126:43013 requested a full tablet report, sending...
I20260809 06:19:27.246196 8632 ts_manager.cc:194] Registered new tserver with Master: 52f0062bdc704ce78deeb5697f3e32f0 (127.3.137.66:40665)
W20260809 06:19:27.564205 8976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:27.564463 8976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:27.564554 8976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:27.571770 8976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:27.571861 8976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:27.571962 8976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:27.580572 8976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:27.581972 8976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:27.583820 8976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:27.596724 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:27.596915 8986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:27.596771 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:27.597709 8976 server_base.cc:1034] running on GCE node
I20260809 06:19:27.598278 8976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:27.599200 8976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:27.600425 8976 hybrid_clock.cc:648] HybridClock initialized: now 1786256367600375 us; error 37 us; skew 500 ppm
I20260809 06:19:27.600728 8976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:27.602880 8976 webserver.cc:460] Webserver started at http://127.3.137.67:33507/ using document root <none> and password file <none>
I20260809 06:19:27.603572 8976 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:27.603704 8976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:27.604031 8976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:27.606778 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "307cc7797fba71fe117c7f73bebe38a9"
server_key_iv: "78be7ae15ed0f93042f28510040d3167"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.607636 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "307cc7797fba71fe117c7f73bebe38a9"
server_key_iv: "78be7ae15ed0f93042f28510040d3167"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.612651 8976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260809 06:19:27.616072 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.617321 8976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:27.617504 8976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72"
format_stamp: "Formatted at 2026-08-09 06:19:27 on dist-test-slave-2w3w"
server_key: "307cc7797fba71fe117c7f73bebe38a9"
server_key_iv: "78be7ae15ed0f93042f28510040d3167"
server_key_version: "encryptionkey@0"
I20260809 06:19:27.617731 8976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:27.623318 8976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:27.624084 8976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:27.624296 8976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:27.625098 8976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:27.626343 8976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:27.626426 8976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.626502 8976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:27.626549 8976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:27.657440 8976 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:46463
I20260809 06:19:27.657492 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:46463 every 8 connection(s)
I20260809 06:19:27.658754 8976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:27.664354 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 8976
I20260809 06:19:27.664510 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:27.664942 3621 external_mini_cluster.cc:1428] Setting key 1a56ed5355905bd43b56555994941283
I20260809 06:19:27.667287 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.661406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:49821 (local address 127.3.137.126:43013)
0809 06:19:27.661631 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.661635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.662887 (+ 1252us) server_negotiation.cc:407] Connection header received
0809 06:19:27.664662 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.664667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.664723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.664783 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.666487 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.666493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.666497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.666516 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.666556 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.666568 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.666698 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.666903 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.666998 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 06:19:27.668157 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.661844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43013 (local address 127.3.137.67:49821)
0809 06:19:27.662720 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.662760 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.664300 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.665221 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.665228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.665312 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.666103 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.666107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.666163 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.666289 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.666667 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.666679 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.666687 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.667010 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.667241 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260809 06:19:27.669417 9115 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43013
I20260809 06:19:27.669696 9115 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:27.670252 9115 heartbeater.cc:511] Master 127.3.137.126:43013 requested a full tablet report, sending...
I20260809 06:19:27.671577 8632 ts_manager.cc:194] Registered new tserver with Master: 2ab8d1b874cb4d7ebb27b15eb8e30c72 (127.3.137.67:46463)
I20260809 06:19:27.680160 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:19:27.687274 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.684751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.3.137.126:43013)
0809 06:19:27.684923 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.684927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.685475 (+ 548us) server_negotiation.cc:407] Connection header received
0809 06:19:27.685612 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.685615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.685662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.685747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.686606 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.686612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.686617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.686636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.686676 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.686688 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.686820 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.686907 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.686997 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:19:27.690913 8632 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:27.698594 8632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:27.723379 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.715562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43226)
0809 06:19:27.716470 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.716519 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.716658 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.720004 (+ 3346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.720012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.720109 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.721009 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.721014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.721070 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.721167 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.722564 (+ 1397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.722579 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.722587 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.722744 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.722849 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":4220,"threads_started":1}
I20260809 06:19:27.724499 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.715747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.3.137.65:45143)
0809 06:19:27.719347 (+ 3600us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.719353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.719377 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:19:27.719432 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.719438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.719717 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.720612 (+ 895us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.722246 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.722255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.722272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.722289 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.722346 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.722366 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.723932 (+ 1566us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.724017 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.724124 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3463,"thread_start_us":3364,"threads_started":1}
I20260809 06:19:27.725140 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.720921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.3.137.67:46463)
0809 06:19:27.721688 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.721696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.721729 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:19:27.721848 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.721855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.722090 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.722251 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.723649 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.723658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.723672 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.723691 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.723751 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.723774 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.723932 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.724636 (+ 704us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.724732 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":441,"threads_started":1}
I20260809 06:19:27.725190 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.720446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:46463 (local address 127.0.0.1:42584)
0809 06:19:27.721220 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.721249 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.721709 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.722246 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.722250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.722274 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.723038 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.723041 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.723058 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.723102 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.723987 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.723992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.723996 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.724131 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.724179 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"spinlock_wait_cycles":744832,"thread_start_us":507,"threads_started":1}
I20260809 06:19:27.725589 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.716270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.3.137.66:40665)
0809 06:19:27.717150 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.717160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.720198 (+ 3038us) server_negotiation.cc:407] Connection header received
0809 06:19:27.721843 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.721854 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.722074 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.722199 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.724667 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.724675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.724688 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.724710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.724772 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.724800 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.724969 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.725150 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.725267 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":568,"threads_started":1}
I20260809 06:19:27.726296 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.715681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40665 (local address 127.0.0.1:59206)
0809 06:19:27.720082 (+ 4401us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.720105 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.721657 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.723604 (+ 1947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.723610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.723632 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.724369 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.724372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.724388 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.724435 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.724921 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.724926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.724931 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.725871 (+ 940us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.725922 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4250,"thread_start_us":408,"threads_started":1}
I20260809 06:19:27.728410 8762 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 (DEFAULT_TABLE table=test-table [id=27009399e33544f8974a48cdbded1bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:27.728960 9045 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 (DEFAULT_TABLE table=test-table [id=27009399e33544f8974a48cdbded1bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:27.729923 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be1de6e2c45ed9b586350e2d235b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:27.730147 9045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be1de6e2c45ed9b586350e2d235b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:27.729873 8903 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 (DEFAULT_TABLE table=test-table [id=27009399e33544f8974a48cdbded1bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:27.731174 8903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be1de6e2c45ed9b586350e2d235b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:27.741422 9135 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Bootstrap starting.
I20260809 06:19:27.744704 9135 tablet_bootstrap.cc:654] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:27.745010 9136 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Bootstrap starting.
I20260809 06:19:27.746394 9135 log.cc:824] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:27.747562 9137 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Bootstrap starting.
I20260809 06:19:27.748744 9136 tablet_bootstrap.cc:654] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:27.750192 9136 log.cc:824] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:27.750532 9137 tablet_bootstrap.cc:654] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:27.751868 9137 log.cc:824] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:27.753091 9135 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: No bootstrap required, opened a new log
I20260809 06:19:27.753114 9136 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: No bootstrap required, opened a new log
I20260809 06:19:27.753392 9136 ts_tablet_manager.cc:1397] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:19:27.753387 9135 ts_tablet_manager.cc:1397] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260809 06:19:27.754818 9137 tablet_bootstrap.cc:492] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: No bootstrap required, opened a new log
I20260809 06:19:27.755169 9137 ts_tablet_manager.cc:1397] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260809 06:19:27.760705 9136 raft_consensus.cc:348] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.761152 9136 raft_consensus.cc:374] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:27.761292 9136 raft_consensus.cc:729] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f0062bdc704ce78deeb5697f3e32f0, State: Initialized, Role: FOLLOWER
I20260809 06:19:27.761914 9136 consensus_queue.cc:260] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.763051 8973 heartbeater.cc:503] Master 127.3.137.126:43013 was elected leader, sending a full tablet report...
I20260809 06:19:27.763636 9136 ts_tablet_manager.cc:1428] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260809 06:19:27.765852 9137 raft_consensus.cc:348] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.765851 9135 raft_consensus.cc:348] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.766271 9137 raft_consensus.cc:374] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:27.766271 9135 raft_consensus.cc:374] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:27.766371 9135 raft_consensus.cc:729] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab8d1b874cb4d7ebb27b15eb8e30c72, State: Initialized, Role: FOLLOWER
I20260809 06:19:27.766459 9137 raft_consensus.cc:729] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff24705ed0c4b8298f446ad26e15411, State: Initialized, Role: FOLLOWER
I20260809 06:19:27.767095 9137 consensus_queue.cc:260] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.767819 9135 consensus_queue.cc:260] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.768255 8831 heartbeater.cc:503] Master 127.3.137.126:43013 was elected leader, sending a full tablet report...
I20260809 06:19:27.769061 9137 ts_tablet_manager.cc:1428] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260809 06:19:27.770102 9115 heartbeater.cc:503] Master 127.3.137.126:43013 was elected leader, sending a full tablet report...
I20260809 06:19:27.771435 9135 ts_tablet_manager.cc:1428] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
W20260809 06:19:27.801734 8832 tablet.cc:2369] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:27.803962 9141 raft_consensus.cc:482] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:27.804241 9141 raft_consensus.cc:504] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.805938 9141 leader_election.cc:288] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff24705ed0c4b8298f446ad26e15411 (127.3.137.65:45143), 2ab8d1b874cb4d7ebb27b15eb8e30c72 (127.3.137.67:46463)
I20260809 06:19:27.809424 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.806211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39591 (local address 127.3.137.65:45143)
0809 06:19:27.806446 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.806456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.807009 (+ 553us) server_negotiation.cc:407] Connection header received
0809 06:19:27.807260 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.807266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.807324 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.807398 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.808675 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.808682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.808690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.808710 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.808767 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.808807 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.808927 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.809054 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.809117 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:27.809631 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.806158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.3.137.66:39591)
0809 06:19:27.806873 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.806899 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.807081 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.807611 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.807616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.807640 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.808438 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.808442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.808460 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.808513 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.808916 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.808921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.808925 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.809090 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.809140 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":347,"threads_started":1}
I20260809 06:19:27.810456 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.807159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:42003 (local address 127.3.137.67:46463)
0809 06:19:27.807336 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.807342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.807608 (+ 266us) server_negotiation.cc:407] Connection header received
0809 06:19:27.807799 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.807804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.807859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.807966 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.809774 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.809780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.809784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.809803 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.809841 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.809858 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.809949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.810190 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.810237 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:19:27.810665 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.806915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:46463 (local address 127.3.137.66:42003)
0809 06:19:27.807502 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:27.807520 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:27.807650 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:27.807993 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:27.807998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:27.808358 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:27.809520 (+ 1162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:19:27.809525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:27.809544 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:19:27.809592 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:27.810051 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:27.810059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:27.810063 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 06:19:27.810201 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 06:19:27.810253 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":318,"threads_started":1}
I20260809 06:19:27.810865 8784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be1de6e2c45ed9b586350e2d235b0" candidate_uuid: "52f0062bdc704ce78deeb5697f3e32f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff24705ed0c4b8298f446ad26e15411" is_pre_election: true
I20260809 06:19:27.811466 8784 raft_consensus.cc:2454] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f0062bdc704ce78deeb5697f3e32f0 in term 0.
I20260809 06:19:27.811595 9068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be1de6e2c45ed9b586350e2d235b0" candidate_uuid: "52f0062bdc704ce78deeb5697f3e32f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" is_pre_election: true
I20260809 06:19:27.812139 8856 leader_election.cc:302] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff24705ed0c4b8298f446ad26e15411, 52f0062bdc704ce78deeb5697f3e32f0; no voters:
I20260809 06:19:27.812142 9068 raft_consensus.cc:2454] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f0062bdc704ce78deeb5697f3e32f0 in term 0.
I20260809 06:19:27.812511 9141 raft_consensus.cc:2781] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:27.812686 9141 raft_consensus.cc:482] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:27.812801 9141 raft_consensus.cc:3037] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:27.815259 9141 raft_consensus.cc:504] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.815908 9141 leader_election.cc:288] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [CANDIDATE]: Term 1 election: Requested vote from peers 2ff24705ed0c4b8298f446ad26e15411 (127.3.137.65:45143), 2ab8d1b874cb4d7ebb27b15eb8e30c72 (127.3.137.67:46463)
I20260809 06:19:27.816265 8784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be1de6e2c45ed9b586350e2d235b0" candidate_uuid: "52f0062bdc704ce78deeb5697f3e32f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff24705ed0c4b8298f446ad26e15411"
I20260809 06:19:27.816357 9068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be1de6e2c45ed9b586350e2d235b0" candidate_uuid: "52f0062bdc704ce78deeb5697f3e32f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72"
I20260809 06:19:27.816504 8784 raft_consensus.cc:3037] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:27.816591 9068 raft_consensus.cc:3037] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:27.818691 8784 raft_consensus.cc:2454] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f0062bdc704ce78deeb5697f3e32f0 in term 1.
I20260809 06:19:27.818841 9068 raft_consensus.cc:2454] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f0062bdc704ce78deeb5697f3e32f0 in term 1.
I20260809 06:19:27.819079 8856 leader_election.cc:302] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff24705ed0c4b8298f446ad26e15411, 52f0062bdc704ce78deeb5697f3e32f0; no voters:
I20260809 06:19:27.819370 9141 raft_consensus.cc:2781] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:27.819860 9141 raft_consensus.cc:686] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 1 LEADER]: Becoming Leader. State: Replica: 52f0062bdc704ce78deeb5697f3e32f0, State: Running, Role: LEADER
I20260809 06:19:27.820379 9141 consensus_queue.cc:237] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } }
I20260809 06:19:27.825212 8632 catalog_manager.cc:5495] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f0062bdc704ce78deeb5697f3e32f0 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "52f0062bdc704ce78deeb5697f3e32f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f0062bdc704ce78deeb5697f3e32f0" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:27.848330 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.846022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.3.137.66:40665)
0809 06:19:27.846227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.846233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.846252 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:19:27.846467 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.846471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.846547 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.846659 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.847619 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.847624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.847629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.847648 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.847684 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.847696 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.847864 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.847945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.848004 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:19:27.856570 9068 raft_consensus.cc:1261] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 1 FOLLOWER]: Refusing update from remote peer 52f0062bdc704ce78deeb5697f3e32f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:27.856570 8784 raft_consensus.cc:1261] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 1 FOLLOWER]: Refusing update from remote peer 52f0062bdc704ce78deeb5697f3e32f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:27.857558 9147 consensus_queue.cc:1035] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff24705ed0c4b8298f446ad26e15411" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:27.857990 9141 consensus_queue.cc:1035] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab8d1b874cb4d7ebb27b15eb8e30c72" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:27.869320 9151 mvcc.cc:205] Tried to move back new op lower bound from 7316506082729410560 to 7316506082596118528. Current Snapshot: MvccSnapshot[applied={T|T < 7316506082729410560}]
I20260809 06:19:27.870849 9153 mvcc.cc:205] Tried to move back new op lower bound from 7316506082729410560 to 7316506082596118528. Current Snapshot: MvccSnapshot[applied={T|T < 7316506082729410560}]
I20260809 06:19:27.872655 9152 mvcc.cc:205] Tried to move back new op lower bound from 7316506082729410560 to 7316506082596118528. Current Snapshot: MvccSnapshot[applied={T|T < 7316506082729410560}]
I20260809 06:19:27.887526 8632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:27.887826 8632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:27.891609 8632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3021cf8db44847b5a3f4ad4b3b689fc8: Sending DeleteTablet for 3 replicas of tablet 1d3be1de6e2c45ed9b586350e2d235b0
I20260809 06:19:27.893002 8903 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:27 UTC) from {username='slave'} at 127.0.0.1:59206
I20260809 06:19:27.893008 8762 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:27 UTC) from {username='slave'} at 127.0.0.1:43226
I20260809 06:19:27.893208 9045 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3be1de6e2c45ed9b586350e2d235b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:27 UTC) from {username='slave'} at 127.0.0.1:42584
I20260809 06:19:27.897042 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8695
I20260809 06:19:27.904019 9164 tablet_replica.cc:330] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: stopping tablet replica
I20260809 06:19:27.904695 9164 raft_consensus.cc:2229] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:27.904892 9165 tablet_replica.cc:330] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: stopping tablet replica
I20260809 06:19:27.905573 9164 raft_consensus.cc:2258] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:27.905553 9165 raft_consensus.cc:2229] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:27.906128 9165 raft_consensus.cc:2258] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:27.907173 9163 tablet_replica.cc:330] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411: stopping tablet replica
I20260809 06:19:27.908560 9164 ts_tablet_manager.cc:1905] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:27.908632 9165 ts_tablet_manager.cc:1905] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:27.912348 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:27.899115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.3.137.65:45143)
0809 06:19:27.899346 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:27.899355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:27.909854 (+ 10499us) server_negotiation.cc:407] Connection header received
0809 06:19:27.910116 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:27.910121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:27.910225 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:27.910315 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:27.911532 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:27.911538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:27.911543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:27.911561 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:27.911596 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:27.911612 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:27.911704 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:27.911971 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 06:19:27.912029 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:19:27.913287 9163 raft_consensus.cc:2229] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:27.913426 8826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:27.916178 9165 ts_tablet_manager.cc:1918] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:27.916412 9165 log.cc:1196] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/1d3be1de6e2c45ed9b586350e2d235b0
I20260809 06:19:27.917068 9165 ts_tablet_manager.cc:1939] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ab8d1b874cb4d7ebb27b15eb8e30c72: Deleting consensus metadata
I20260809 06:19:27.918985 9164 ts_tablet_manager.cc:1918] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:27.919252 9164 log.cc:1196] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/1d3be1de6e2c45ed9b586350e2d235b0
I20260809 06:19:27.919291 8618 catalog_manager.cc:4841] TS 2ab8d1b874cb4d7ebb27b15eb8e30c72 (127.3.137.67:46463): tablet 1d3be1de6e2c45ed9b586350e2d235b0 (table test-table [id=27009399e33544f8974a48cdbded1bbb]) successfully deleted
I20260809 06:19:27.919967 9164 ts_tablet_manager.cc:1939] T 1d3be1de6e2c45ed9b586350e2d235b0 P 52f0062bdc704ce78deeb5697f3e32f0: Deleting consensus metadata
I20260809 06:19:27.921849 8614 catalog_manager.cc:4841] TS 52f0062bdc704ce78deeb5697f3e32f0 (127.3.137.66:40665): tablet 1d3be1de6e2c45ed9b586350e2d235b0 (table test-table [id=27009399e33544f8974a48cdbded1bbb]) successfully deleted
I20260809 06:19:28.055740 9163 raft_consensus.cc:2258] T 1d3be1de6e2c45ed9b586350e2d235b0 P 2ff24705ed0c4b8298f446ad26e15411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:28.056067 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8695
I20260809 06:19:28.081115 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8835
W20260809 06:19:28.081082 8618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:45143 (error 108)
I20260809 06:19:28.083343 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.082802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43254)
0809 06:19:28.082989 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.083141 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260809 06:19:28.083614 8618 catalog_manager.cc:4584] TS 2ff24705ed0c4b8298f446ad26e15411 (127.3.137.65:45143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d3be1de6e2c45ed9b586350e2d235b0: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
I20260809 06:19:28.084915 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.082146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.3.137.66:40665)
0809 06:19:28.082338 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:28.082344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:28.082368 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:19:28.083084 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:28.083094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:28.083164 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:28.083272 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:28.084247 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:28.084252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:28.084255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:28.084269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:28.084299 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:28.084310 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:28.084413 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:28.084542 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 06:19:28.084588 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:19:28.085716 8967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:28.137264 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.136739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43262)
0809 06:19:28.137005 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.137098 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 06:19:28.138337 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.137982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43276)
0809 06:19:28.138127 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.138221 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 06:19:28.193177 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8835
I20260809 06:19:28.199175 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.198518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43280)
0809 06:19:28.198791 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.198927 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 06:19:28.200289 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.199841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43288)
0809 06:19:28.200023 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.200128 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 06:19:28.212637 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8976
I20260809 06:19:28.215529 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.213423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.3.137.67:46463)
0809 06:19:28.213586 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:28.213591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:28.213608 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:28.213780 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:28.213784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:28.213859 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:28.213940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:28.214926 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:28.214930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:28.214933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:28.214945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:28.215015 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:28.215035 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:28.215143 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:28.215181 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 06:19:28.215223 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:19:28.216176 9110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:28.280766 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.280225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43294)
0809 06:19:28.280471 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.280592 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:19:28.281987 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.281506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45143 (local address 127.0.0.1:43298)
0809 06:19:28.281759 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:28.281845 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260809 06:19:28.303983 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8976
I20260809 06:19:28.322752 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 8598
I20260809 06:19:28.323984 8665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:28.430533 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 8598
2026-08-09T06:19:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2635 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 06:19:28.473951 3621 mini_oidc.cc:152] Starting JWKS server
I20260809 06:19:28.476635 3621 webserver.cc:460] Webserver started at https://127.0.0.1:39969/ using document root <none> and password file <none>
I20260809 06:19:28.476974 3621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:19:28.477612 3621 webserver.cc:460] Webserver started at http://127.0.0.1:37767/ using document root <none> and password file <none>
2026-08-09T06:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:28Z Disabled control of system clock
I20260809 06:19:28.507848 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45651
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45651
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/testchainca.cert with env {}
W20260809 06:19:28.826925 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:28.827227 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:28.827315 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:28.834362 9177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:28.834465 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:28.834502 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:28.834550 9177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:28.834597 9177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:19:28.834658 9177 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:19:28.834698 9177 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39969/jwks.json
I20260809 06:19:28.844349 9177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:28.845521 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:28.846719 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:28.858405 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:28.858306 9186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:28.858345 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:28.859098 9177 server_base.cc:1034] running on GCE node
I20260809 06:19:28.859898 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:28.861209 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:28.862442 9177 hybrid_clock.cc:648] HybridClock initialized: now 1786256368862399 us; error 57 us; skew 500 ppm
I20260809 06:19:28.862787 9177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:28.865087 9177 webserver.cc:460] Webserver started at http://127.3.137.126:36105/ using document root <none> and password file <none>
I20260809 06:19:28.865741 9177 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:28.865885 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:28.866201 9177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:28.869132 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d"
format_stamp: "Formatted at 2026-08-09 06:19:28 on dist-test-slave-2w3w"
server_key: "fb3d4d05963f0be65c4ca286b4d27f99"
server_key_iv: "2205a63543f3d4df22d65abece307346"
server_key_version: "encryptionkey@0"
I20260809 06:19:28.870018 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d"
format_stamp: "Formatted at 2026-08-09 06:19:28 on dist-test-slave-2w3w"
server_key: "fb3d4d05963f0be65c4ca286b4d27f99"
server_key_iv: "2205a63543f3d4df22d65abece307346"
server_key_version: "encryptionkey@0"
I20260809 06:19:28.875116 9177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:19:28.878348 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:28.879704 9177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:28.879892 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d"
format_stamp: "Formatted at 2026-08-09 06:19:28 on dist-test-slave-2w3w"
server_key: "fb3d4d05963f0be65c4ca286b4d27f99"
server_key_iv: "2205a63543f3d4df22d65abece307346"
server_key_version: "encryptionkey@0"
I20260809 06:19:28.880116 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:28.904544 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:28.905432 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:28.905751 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:28.924149 9177 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45651
I20260809 06:19:28.924156 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45651 every 8 connection(s)
I20260809 06:19:28.925706 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:28.929524 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:28.935823 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9177
I20260809 06:19:28.936058 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:28.936688 3621 external_mini_cluster.cc:1428] Setting key d117672fbc1521cc766688ac9ef855b3
I20260809 06:19:28.941444 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d: Bootstrap starting.
I20260809 06:19:28.945992 9248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:28.947393 9248 log.cc:824] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:28.951025 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d: No bootstrap required, opened a new log
I20260809 06:19:28.954090 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:28.938862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.3.137.126:45651)
0809 06:19:28.940087 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:28.940114 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:28.940170 (+ 56us) server_negotiation.cc:407] Connection header received
0809 06:19:28.942838 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:28.942868 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:28.943562 (+ 694us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:28.944047 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:28.944638 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:28.946892 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:28.948568 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:28.949626 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:28.949817 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:28.951053 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:28.951067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:28.951088 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:28.951884 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:28.951996 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:28.952025 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:28.952152 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:28.952368 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 06:19:28.952701 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":352,"threads_started":1}
I20260809 06:19:28.958148 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:19:28.961001 9248 raft_consensus.cc:348] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } }
I20260809 06:19:28.961450 9248 raft_consensus.cc:374] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:28.961565 9248 raft_consensus.cc:729] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e074f7a082ce4aa3b3aa0cfce2a5ef0d, State: Initialized, Role: FOLLOWER
I20260809 06:19:28.962108 9248 consensus_queue.cc:260] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } }
I20260809 06:19:28.962358 9248 raft_consensus.cc:388] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:28.962499 9248 raft_consensus.cc:482] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:28.962689 9248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:28.965268 9248 raft_consensus.cc:504] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } }
I20260809 06:19:28.965785 9248 leader_election.cc:302] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e074f7a082ce4aa3b3aa0cfce2a5ef0d; no voters:
I20260809 06:19:28.966461 9248 leader_election.cc:288] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:28.966636 9253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:28.967450 9253 raft_consensus.cc:686] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [term 1 LEADER]: Becoming Leader. State: Replica: e074f7a082ce4aa3b3aa0cfce2a5ef0d, State: Running, Role: LEADER
I20260809 06:19:28.968047 9253 consensus_queue.cc:237] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } }
I20260809 06:19:28.968652 9248 sys_catalog.cc:564] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:28.971762 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e074f7a082ce4aa3b3aa0cfce2a5ef0d. Latest consensus state: current_term: 1 leader_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } } }
I20260809 06:19:28.972211 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:28.972376 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e074f7a082ce4aa3b3aa0cfce2a5ef0d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45651 } } }
I20260809 06:19:28.972747 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:28.974267 9262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:28.979406 9262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:28.988543 9262 catalog_manager.cc:1347] Generated new cluster ID: 84262acfd0cf4e519f1a12751ee02f6c
I20260809 06:19:28.988675 9262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:29.006204 9262 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:29.007622 9262 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:29.015277 9262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e074f7a082ce4aa3b3aa0cfce2a5ef0d: Generated new TSK 0
I20260809 06:19:29.016161 9262 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 06:19:29.055351 9279 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 06:19:29.055665 9279 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:19:29.055699 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:29.030758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.3.137.126:45651)
0809 06:19:29.030938 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:29.030943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:29.031508 (+ 565us) server_negotiation.cc:407] Connection header received
0809 06:19:29.031916 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:29.031920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:29.031995 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:29.032140 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:19:29.032721 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.033450 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.034885 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.035449 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.035590 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:29.036149 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:19:29.054999 (+ 18850us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 06:19:29.055385 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 06:19:29.055883 9251 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 06:19:29.057675 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:29.030736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45651 (local address 127.0.0.1:55008)
0809 06:19:29.031395 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:29.031412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:29.031610 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:29.032185 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:29.032191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:29.032204 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:19:29.032551 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:29.032560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.033604 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.033610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:29.034729 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:29.034741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.035639 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.035647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:29.035788 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:29.035844 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:19:29.055355 (+ 19511us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 06:19:29.055730 (+ 375us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.137.126:45651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":361,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.137.126:45651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 06:19:29.059286 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9177
I20260809 06:19:29.060557 9242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:29.154130 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9177
2026-08-09T06:19:29Z chronyd exiting
I20260809 06:19:29.202903 3621 test_util.cc:175] -----------------------------------------------
I20260809 06:19:29.203091 3621 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786256324325922-3621-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 06:19:29.208396 3621 mini_oidc.cc:152] Starting JWKS server
I20260809 06:19:29.210268 3621 webserver.cc:460] Webserver started at https://127.0.0.1:38165/ using document root <none> and password file <none>
I20260809 06:19:29.210515 3621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:19:29.211236 3621 webserver.cc:460] Webserver started at http://127.0.0.1:39151/ using document root <none> and password file <none>
2026-08-09T06:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:29Z Disabled control of system clock
I20260809 06:19:29.240756 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33231
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:33231
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38165/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/testchainca.cert with env {}
W20260809 06:19:29.563611 9286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:29.563932 9286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:29.564040 9286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:29.571638 9286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:29.571748 9286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:29.571794 9286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:29.571858 9286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:29.571903 9286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:19:29.571959 9286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:19:29.572000 9286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38165/jwks.json
I20260809 06:19:29.582561 9286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:33231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38165/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:29.584151 9286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:29.585475 9286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:29.597766 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:29.598022 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:29.597867 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:29.598760 9286 server_base.cc:1034] running on GCE node
I20260809 06:19:29.599649 9286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:29.601018 9286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:29.602221 9286 hybrid_clock.cc:648] HybridClock initialized: now 1786256369602185 us; error 31 us; skew 500 ppm
I20260809 06:19:29.602566 9286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:29.611584 9286 webserver.cc:460] Webserver started at http://127.3.137.126:36153/ using document root <none> and password file <none>
I20260809 06:19:29.612329 9286 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:29.612506 9286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:29.612805 9286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:29.615638 9286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "dc2f9d2ea0c040c1863e9c67356c97c1"
format_stamp: "Formatted at 2026-08-09 06:19:29 on dist-test-slave-2w3w"
server_key: "d80fdb4da6b5e7aaab06a7d364664442"
server_key_iv: "b65a6167c38195b2b9aea4ecf76ec561"
server_key_version: "encryptionkey@0"
I20260809 06:19:29.616364 9286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "dc2f9d2ea0c040c1863e9c67356c97c1"
format_stamp: "Formatted at 2026-08-09 06:19:29 on dist-test-slave-2w3w"
server_key: "d80fdb4da6b5e7aaab06a7d364664442"
server_key_iv: "b65a6167c38195b2b9aea4ecf76ec561"
server_key_version: "encryptionkey@0"
I20260809 06:19:29.621318 9286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:19:29.624751 9303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:29.626173 9286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:29.626387 9286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "dc2f9d2ea0c040c1863e9c67356c97c1"
format_stamp: "Formatted at 2026-08-09 06:19:29 on dist-test-slave-2w3w"
server_key: "d80fdb4da6b5e7aaab06a7d364664442"
server_key_iv: "b65a6167c38195b2b9aea4ecf76ec561"
server_key_version: "encryptionkey@0"
I20260809 06:19:29.626616 9286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:29.640272 9286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:29.641099 9286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:29.641398 9286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:29.659775 9286 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:33231
I20260809 06:19:29.659807 9357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:33231 every 8 connection(s)
I20260809 06:19:29.661125 9286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:29.664757 9358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:29.669811 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9286
I20260809 06:19:29.670012 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:29.670677 3621 external_mini_cluster.cc:1428] Setting key f225f1678c9fcd80812c8df94e4c6e68
I20260809 06:19:29.676733 9358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1: Bootstrap starting.
I20260809 06:19:29.680656 9358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:29.681810 9358 log.cc:824] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:29.684868 9358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1: No bootstrap required, opened a new log
I20260809 06:19:29.686550 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:29.672817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.3.137.126:33231)
0809 06:19:29.673911 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:29.673924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:29.673981 (+ 57us) server_negotiation.cc:407] Connection header received
0809 06:19:29.676461 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:29.676488 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:29.677044 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:29.677473 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:29.677985 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.680035 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.681332 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.682396 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.682584 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:29.684225 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:29.684238 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:29.684253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:29.684804 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:29.684884 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:29.684904 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:29.685042 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:29.685316 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 06:19:29.685631 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":350,"threads_started":1}
I20260809 06:19:29.690369 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:19:29.692894 9358 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } }
I20260809 06:19:29.693269 9358 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:29.693372 9358 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2f9d2ea0c040c1863e9c67356c97c1, State: Initialized, Role: FOLLOWER
I20260809 06:19:29.694023 9358 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } }
I20260809 06:19:29.694271 9358 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:29.694427 9358 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:29.694605 9358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:29.696841 9358 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } }
I20260809 06:19:29.697332 9358 leader_election.cc:302] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2f9d2ea0c040c1863e9c67356c97c1; no voters:
I20260809 06:19:29.697909 9358 leader_election.cc:288] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:29.698083 9363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:29.698853 9363 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [term 1 LEADER]: Becoming Leader. State: Replica: dc2f9d2ea0c040c1863e9c67356c97c1, State: Running, Role: LEADER
I20260809 06:19:29.699481 9363 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } }
I20260809 06:19:29.700219 9358 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:29.702663 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } } }
I20260809 06:19:29.702863 9365 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2f9d2ea0c040c1863e9c67356c97c1. Latest consensus state: current_term: 1 leader_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f9d2ea0c040c1863e9c67356c97c1" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33231 } } }
I20260809 06:19:29.703212 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:29.703210 9365 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:29.704159 9372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:29.710099 9372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:29.719597 9372 catalog_manager.cc:1347] Generated new cluster ID: 8b6ca082999e4c4cab628b4bcac579fe
I20260809 06:19:29.719709 9372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:29.736851 9372 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:29.738133 9372 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:29.745637 9372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc2f9d2ea0c040c1863e9c67356c97c1: Generated new TSK 0
I20260809 06:19:29.746685 9372 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 06:19:29.780833 9389 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 06:19:29.780965 9389 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:19:29.782123 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:29.761778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.3.137.126:33231)
0809 06:19:29.761981 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:29.761987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:29.762589 (+ 602us) server_negotiation.cc:407] Connection header received
0809 06:19:29.762824 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:29.762830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:29.762920 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:29.763112 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:19:29.763576 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.764687 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.766043 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.766749 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.766888 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:29.767427 (+ 539us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:19:29.781509 (+ 14082us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 06:19:29.781841 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 06:19:29.782158 9388 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:29.761626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33231 (local address 127.0.0.1:55984)
0809 06:19:29.762479 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:29.762496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:29.762639 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:29.763135 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:29.763138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:29.763149 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:19:29.763421 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:29.763429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.764841 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.764845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:29.765891 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:29.765902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:29.766920 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:29.766924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:29.767128 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:29.767162 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:19:29.781708 (+ 14546us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 06:19:29.781893 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.137.126:33231: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":536,"threads_started":1}
W20260809 06:19:29.782244 9361 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 06:19:29.783684 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9286
I20260809 06:19:29.784816 9353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:29.880191 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9286
2026-08-09T06:19:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (730 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 06:19:29.938123 3621 mini_oidc.cc:152] Starting JWKS server
I20260809 06:19:29.940199 3621 webserver.cc:460] Webserver started at https://127.0.0.1:43141/ using document root <none> and password file <none>
I20260809 06:19:29.940505 3621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:19:29.941161 3621 webserver.cc:460] Webserver started at http://127.0.0.1:35007/ using document root <none> and password file <none>
2026-08-09T06:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:29Z Disabled control of system clock
I20260809 06:19:29.973259 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35905
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:35905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 06:19:30.297672 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:30.298012 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:30.298166 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:30.305310 9397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:30.305462 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:30.305545 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:30.305660 9397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:30.305739 9397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:19:30.305827 9397 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:19:30.305905 9397 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43141/jwks.json
I20260809 06:19:30.315030 9397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:35905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:30.316324 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:30.318158 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:30.329984 9403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:30.330017 9402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:30.330065 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:30.330864 9397 server_base.cc:1034] running on GCE node
I20260809 06:19:30.331692 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:30.333109 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:30.334344 9397 hybrid_clock.cc:648] HybridClock initialized: now 1786256370334315 us; error 55 us; skew 500 ppm
I20260809 06:19:30.334692 9397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:30.337031 9397 webserver.cc:460] Webserver started at http://127.3.137.126:37135/ using document root <none> and password file <none>
I20260809 06:19:30.337728 9397 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:30.337862 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:30.338213 9397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:30.341158 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "45230250f5024bfd95d078131f572f87"
format_stamp: "Formatted at 2026-08-09 06:19:30 on dist-test-slave-2w3w"
server_key: "7ef8de0c414d914e38a3abff6f44a1b9"
server_key_iv: "7c459f8b16c929a9e8ac06aeead95e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:30.342022 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "45230250f5024bfd95d078131f572f87"
format_stamp: "Formatted at 2026-08-09 06:19:30 on dist-test-slave-2w3w"
server_key: "7ef8de0c414d914e38a3abff6f44a1b9"
server_key_iv: "7c459f8b16c929a9e8ac06aeead95e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:30.347069 9397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 06:19:30.350325 9412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:30.351648 9397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:19:30.351833 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "45230250f5024bfd95d078131f572f87"
format_stamp: "Formatted at 2026-08-09 06:19:30 on dist-test-slave-2w3w"
server_key: "7ef8de0c414d914e38a3abff6f44a1b9"
server_key_iv: "7c459f8b16c929a9e8ac06aeead95e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:30.352061 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:30.369405 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:30.370163 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:30.370481 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:30.388209 9397 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:35905
I20260809 06:19:30.388242 9468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:35905 every 8 connection(s)
I20260809 06:19:30.389550 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:30.392023 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9397
I20260809 06:19:30.392244 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:30.393065 3621 external_mini_cluster.cc:1428] Setting key 54d2f4266b67bb64128981d5456e8b93
I20260809 06:19:30.393574 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:30.406776 9469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87: Bootstrap starting.
I20260809 06:19:30.407234 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:30.394926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.3.137.126:35905)
0809 06:19:30.395958 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:30.395968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:30.396012 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:19:30.397972 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:30.397994 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:30.398435 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:30.398748 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:30.399395 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.400783 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.402610 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.403495 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.403632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:30.404894 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:30.404901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:30.404913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:30.405460 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:30.405532 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:30.405552 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:30.405637 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:30.405876 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 06:19:30.406204 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":398,"threads_started":1}
I20260809 06:19:30.410251 9469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:30.411221 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:19:30.411592 9469 log.cc:824] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:30.414640 9469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87: No bootstrap required, opened a new log
I20260809 06:19:30.422468 9469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } }
I20260809 06:19:30.422802 9469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:30.422873 9469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45230250f5024bfd95d078131f572f87, State: Initialized, Role: FOLLOWER
I20260809 06:19:30.423393 9469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } }
I20260809 06:19:30.423586 9469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:30.423681 9469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:30.423795 9469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:30.425798 9469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } }
I20260809 06:19:30.426242 9469 leader_election.cc:302] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45230250f5024bfd95d078131f572f87; no voters:
I20260809 06:19:30.426759 9469 leader_election.cc:288] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:30.426930 9474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:30.427785 9474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [term 1 LEADER]: Becoming Leader. State: Replica: 45230250f5024bfd95d078131f572f87, State: Running, Role: LEADER
I20260809 06:19:30.428318 9474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } }
I20260809 06:19:30.428943 9469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:30.431062 9476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45230250f5024bfd95d078131f572f87. Latest consensus state: current_term: 1 leader_uuid: "45230250f5024bfd95d078131f572f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } } }
I20260809 06:19:30.431423 9476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:30.431275 9475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45230250f5024bfd95d078131f572f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45230250f5024bfd95d078131f572f87" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35905 } } }
I20260809 06:19:30.431715 9475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:30.433350 9479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:30.438460 9479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:30.447201 9479 catalog_manager.cc:1347] Generated new cluster ID: 2deb6deb4214436b81f6f20481b1120b
I20260809 06:19:30.447324 9479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:30.464048 9479 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:30.465268 9479 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:30.471980 9479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45230250f5024bfd95d078131f572f87: Generated new TSK 0
I20260809 06:19:30.472766 9479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:19:30.496943 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:30.488367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.3.137.126:35905)
0809 06:19:30.488561 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:30.488568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:30.488963 (+ 395us) server_negotiation.cc:407] Connection header received
0809 06:19:30.489167 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:30.489173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:30.489282 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:30.489445 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:19:30.489858 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.491047 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.492408 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.493103 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.493225 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:30.493759 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:19:30.496151 (+ 2392us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 06:19:30.496626 (+ 475us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:19:30.497078 9472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 06:19:30.498525 9500 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:30.488240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:35905 (local address 127.0.0.1:42882)
0809 06:19:30.488860 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:30.488876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:30.488997 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:30.489461 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:30.489464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:30.489474 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:19:30.489711 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:30.489718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.491179 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.491183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:30.492256 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:30.492267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:30.493283 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:30.493287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:30.493409 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:30.493445 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:19:30.496511 (+ 3066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 06:19:30.498112 (+ 1601us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.137.126:35905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1304,"thread_start_us":335,"threads_started":1}
I20260809 06:19:30.499588 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9397
I20260809 06:19:30.500725 9463 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 06:19:30.508612 9501 webserver.cc:533] Webserver: sslize failed:
E20260809 06:19:30.508697 9501 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:19:30.592890 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9397
2026-08-09T06:19:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (711 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T06:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:30Z Disabled control of system clock
I20260809 06:19:30.681444 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43859
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43859
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/testchainca.cert with env {}
W20260809 06:19:31.009187 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:31.009497 9506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:31.009588 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:31.016837 9506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:31.016947 9506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:31.016986 9506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:31.017043 9506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:31.017088 9506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:19:31.017148 9506 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:19:31.017189 9506 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 06:19:31.027021 9506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:31.028239 9506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:31.029465 9506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:31.041272 9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:31.041236 9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:31.041318 9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:31.042347 9506 server_base.cc:1034] running on GCE node
I20260809 06:19:31.043082 9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:31.044330 9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:31.045552 9506 hybrid_clock.cc:648] HybridClock initialized: now 1786256371045512 us; error 40 us; skew 500 ppm
I20260809 06:19:31.046049 9506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:31.049311 9506 webserver.cc:460] Webserver started at http://127.3.137.126:42625/ using document root <none> and password file <none>
I20260809 06:19:31.050293 9506 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:31.050422 9506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:31.050860 9506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:31.054085 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "2b5229fd871e4d52bdedbf8c5d2b680c"
format_stamp: "Formatted at 2026-08-09 06:19:31 on dist-test-slave-2w3w"
server_key: "be64762a0146e4def7c825b3ec702665"
server_key_iv: "b4fb147d0b87c389bf9daae8ab02cef0"
server_key_version: "encryptionkey@0"
I20260809 06:19:31.054996 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "2b5229fd871e4d52bdedbf8c5d2b680c"
format_stamp: "Formatted at 2026-08-09 06:19:31 on dist-test-slave-2w3w"
server_key: "be64762a0146e4def7c825b3ec702665"
server_key_iv: "b4fb147d0b87c389bf9daae8ab02cef0"
server_key_version: "encryptionkey@0"
I20260809 06:19:31.060145 9506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:19:31.063409 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:31.064622 9506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:19:31.064808 9506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "2b5229fd871e4d52bdedbf8c5d2b680c"
format_stamp: "Formatted at 2026-08-09 06:19:31 on dist-test-slave-2w3w"
server_key: "be64762a0146e4def7c825b3ec702665"
server_key_iv: "b4fb147d0b87c389bf9daae8ab02cef0"
server_key_version: "encryptionkey@0"
I20260809 06:19:31.065016 9506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:31.080418 9506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:31.081194 9506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:31.081476 9506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:31.099658 9506 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:43859
I20260809 06:19:31.099692 9572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:43859 every 8 connection(s)
I20260809 06:19:31.101001 9506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:31.102887 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9506
I20260809 06:19:31.103109 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:31.103772 3621 external_mini_cluster.cc:1428] Setting key 944e5c002b6ccef4dde20f99c65a0c4f
I20260809 06:19:31.104960 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:31.116966 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c: Bootstrap starting.
I20260809 06:19:31.117223 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:31.105655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.3.137.126:43859)
0809 06:19:31.106631 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:31.106642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:31.106682 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:19:31.108361 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:31.108382 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:31.108832 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:31.109263 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:31.109855 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.111294 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.112640 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.113325 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.113525 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:31.114812 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:31.114820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:31.114833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:19:31.115472 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:31.115546 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:19:31.115567 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:31.115699 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:31.115935 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 06:19:31.116244 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260809 06:19:31.120673 9573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:31.121097 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:19:31.121716 9573 log.cc:824] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:31.124461 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c: No bootstrap required, opened a new log
I20260809 06:19:31.131552 9573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } }
I20260809 06:19:31.131924 9573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:31.132047 9573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5229fd871e4d52bdedbf8c5d2b680c, State: Initialized, Role: FOLLOWER
I20260809 06:19:31.132676 9573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } }
I20260809 06:19:31.132927 9573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:31.133086 9573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:31.133271 9573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:31.135457 9573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } }
I20260809 06:19:31.135946 9573 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5229fd871e4d52bdedbf8c5d2b680c; no voters:
I20260809 06:19:31.136521 9573 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:31.136677 9578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:31.137444 9578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [term 1 LEADER]: Becoming Leader. State: Replica: 2b5229fd871e4d52bdedbf8c5d2b680c, State: Running, Role: LEADER
I20260809 06:19:31.138037 9578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } }
I20260809 06:19:31.138688 9573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:31.140990 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } } }
I20260809 06:19:31.141036 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5229fd871e4d52bdedbf8c5d2b680c. Latest consensus state: current_term: 1 leader_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5229fd871e4d52bdedbf8c5d2b680c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43859 } } }
I20260809 06:19:31.141498 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:31.141499 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:31.143142 9584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:31.147933 9584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:31.157750 9584 catalog_manager.cc:1347] Generated new cluster ID: fc8bf574884d41d6ae69a3b1dee86379
I20260809 06:19:31.157873 9584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:31.185124 9584 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:31.187003 9584 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:31.194329 9584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5229fd871e4d52bdedbf8c5d2b680c: Generated new TSK 0
I20260809 06:19:31.195393 9584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:19:31.241207 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:31.218918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.3.137.126:43859)
0809 06:19:31.219161 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:31.219169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:31.219701 (+ 532us) server_negotiation.cc:407] Connection header received
0809 06:19:31.219965 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:31.219969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:31.220037 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:31.220172 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:19:31.220707 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.221448 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.223488 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.223942 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.224037 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:31.224592 (+ 555us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:19:31.240665 (+ 16073us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 06:19:31.240976 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43894: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 06:19:31.241376 9576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43894: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 06:19:31.241678 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:31.218916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43859 (local address 127.0.0.1:43894)
0809 06:19:31.219583 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:31.219603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:31.219756 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:31.220210 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:31.220215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:31.220229 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:19:31.220558 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:31.220570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.221597 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.221604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:31.223354 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:31.223369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:31.224111 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:31.224118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:31.224313 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:31.224359 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:19:31.240941 (+ 16582us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 06:19:31.241235 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.137.126:43859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":310,"threads_started":1}
I20260809 06:19:31.242758 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9506
I20260809 06:19:31.244082 9568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:31.340281 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9506
2026-08-09T06:19:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (727 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (415 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:32 dist-test-slave-2w3w krb5kdc[9622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:32 dist-test-slave-2w3w krb5kdc[9622](info): set up 2 sockets
Aug 09 06:19:32 dist-test-slave-2w3w krb5kdc[9622](info): commencing operation
krb5kdc: starting...
W20260809 06:19:33.296783 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:33 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:33.522367 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:38645
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:33.849371 9636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:33.849673 9636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:33.849762 9636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:33.856798 9636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:33.856909 9636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:33.856956 9636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:33.857013 9636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:33.857056 9636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:33.866596 9636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:33.867878 9636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:33.869174 9636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:33.880568 9644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:33.880661 9641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:33.880635 9642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:33.881640 9636 server_base.cc:1034] running on GCE node
I20260809 06:19:33.882457 9636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:33.883899 9636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:33.885109 9636 hybrid_clock.cc:648] HybridClock initialized: now 1786256373885057 us; error 41 us; skew 500 ppm
I20260809 06:19:33.885466 9636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:33 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:33.888593 9636 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:33.890034 9636 webserver.cc:460] Webserver started at http://127.3.137.126:36413/ using document root <none> and password file <none>
I20260809 06:19:33.890702 9636 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:33.890829 9636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:33.891212 9636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:33.894114 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "44704beeaa48453884c6841d0d8b4a0c"
format_stamp: "Formatted at 2026-08-09 06:19:33 on dist-test-slave-2w3w"
server_key: "77e1667705b41c8709b0e97f8d19c8b7"
server_key_iv: "1ec2a38c818462c3aa8396b252f05e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:33.894912 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "44704beeaa48453884c6841d0d8b4a0c"
format_stamp: "Formatted at 2026-08-09 06:19:33 on dist-test-slave-2w3w"
server_key: "77e1667705b41c8709b0e97f8d19c8b7"
server_key_iv: "1ec2a38c818462c3aa8396b252f05e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:33.900080 9636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:33.903515 9652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:33.904778 9636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:19:33.904950 9636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "44704beeaa48453884c6841d0d8b4a0c"
format_stamp: "Formatted at 2026-08-09 06:19:33 on dist-test-slave-2w3w"
server_key: "77e1667705b41c8709b0e97f8d19c8b7"
server_key_iv: "1ec2a38c818462c3aa8396b252f05e1d"
server_key_version: "encryptionkey@0"
I20260809 06:19:33.905145 9636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:33.939359 9636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:33.943583 9636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:33.943920 9636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:33.962724 9636 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:38645
I20260809 06:19:33.962765 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:38645 every 8 connection(s)
I20260809 06:19:33.964144 9636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:33.967876 9706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:33.974020 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9636
I20260809 06:19:33.974247 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:33.975135 3621 external_mini_cluster.cc:1428] Setting key 5dcb4c5d2f9e36ad239ac355a733e29d
I20260809 06:19:33.979261 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c: Bootstrap starting.
I20260809 06:19:33.983150 9706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:33.984352 9706 log.cc:824] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:33.987741 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c: No bootstrap required, opened a new log
Aug 09 06:19:33 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:33.998473 9706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } }
I20260809 06:19:33.999006 9706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:33.999128 9706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44704beeaa48453884c6841d0d8b4a0c, State: Initialized, Role: FOLLOWER
I20260809 06:19:33.999835 9706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } }
I20260809 06:19:34.000130 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:33.977237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.3.137.126:38645)
0809 06:19:33.978029 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:33.978040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:33.978082 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:19:33.979861 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:33.979884 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:33.980259 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:33.980559 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:33.982413 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:33.983882 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:33.985637 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:33.986294 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:33.986455 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:33.991294 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:33.991311 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:33.991343 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:33.991406 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:33.995549 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:33.996200 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:33.996209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:33.996218 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:33.996442 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:33.996791 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:33.996796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:33.996799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:33.997536 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:33.997825 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:33.998198 (+ 373us) server_negotiation.cc:299] Negotiation successful
0809 06:19:33.998629 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":372,"threads_started":1}
I20260809 06:19:34.000131 9706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:34.000309 9706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:34.000551 9706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:34.003096 9706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } }
I20260809 06:19:34.003579 9706 leader_election.cc:302] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44704beeaa48453884c6841d0d8b4a0c; no voters:
I20260809 06:19:34.004220 9706 leader_election.cc:288] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:34.004385 9713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:34.005112 9713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [term 1 LEADER]: Becoming Leader. State: Replica: 44704beeaa48453884c6841d0d8b4a0c, State: Running, Role: LEADER
I20260809 06:19:34.005724 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } }
I20260809 06:19:34.006532 9706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:34.009193 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44704beeaa48453884c6841d0d8b4a0c. Latest consensus state: current_term: 1 leader_uuid: "44704beeaa48453884c6841d0d8b4a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } } }
I20260809 06:19:34.009173 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44704beeaa48453884c6841d0d8b4a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44704beeaa48453884c6841d0d8b4a0c" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 38645 } } }
I20260809 06:19:34.009685 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:34.009702 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:34.011034 9722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:34.017026 9722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:34.027935 9722 catalog_manager.cc:1347] Generated new cluster ID: ddb87ba52a3940009c1997f27dc3fd5a
I20260809 06:19:34.028070 9722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:34.042357 9722 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:34.043696 9722 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:34.050860 9722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44704beeaa48453884c6841d0d8b4a0c: Generated new TSK 0
I20260809 06:19:34.051889 9722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:34.104099 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:34.436281 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:34.436549 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:34.436642 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:34.443660 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:34.443763 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:34.443871 9736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:34.453541 9736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:34.454779 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:34.456151 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:34.468730 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:34.468688 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:34.468806 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:34.469838 9736 server_base.cc:1034] running on GCE node
I20260809 06:19:34.470444 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:34.471387 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:34.472627 9736 hybrid_clock.cc:648] HybridClock initialized: now 1786256374472591 us; error 46 us; skew 500 ppm
I20260809 06:19:34.472944 9736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:34 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:34.476429 9736 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:34.477890 9736 webserver.cc:460] Webserver started at http://127.3.137.65:38481/ using document root <none> and password file <none>
I20260809 06:19:34.478564 9736 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:34.478703 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:34.479106 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:34.482028 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "c9643e9173f04da6ae93503cbe91d7d0"
format_stamp: "Formatted at 2026-08-09 06:19:34 on dist-test-slave-2w3w"
server_key: "c0ec35a142b8960b21f49bf3b6d7e050"
server_key_iv: "24427051d4cbdbc21904855daa1c1b90"
server_key_version: "encryptionkey@0"
I20260809 06:19:34.482882 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "c9643e9173f04da6ae93503cbe91d7d0"
format_stamp: "Formatted at 2026-08-09 06:19:34 on dist-test-slave-2w3w"
server_key: "c0ec35a142b8960b21f49bf3b6d7e050"
server_key_iv: "24427051d4cbdbc21904855daa1c1b90"
server_key_version: "encryptionkey@0"
I20260809 06:19:34.488257 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:34.491678 9753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:34.492981 9736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:34.493162 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "c9643e9173f04da6ae93503cbe91d7d0"
format_stamp: "Formatted at 2026-08-09 06:19:34 on dist-test-slave-2w3w"
server_key: "c0ec35a142b8960b21f49bf3b6d7e050"
server_key_iv: "24427051d4cbdbc21904855daa1c1b90"
server_key_version: "encryptionkey@0"
I20260809 06:19:34.493374 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:34.521286 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:34.525686 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:34.525993 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:34.527040 9736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:34.528405 9736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:34.528517 9736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:34.528664 9736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:34.528735 9736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:34.562549 9736 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:46841
I20260809 06:19:34.562636 9877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:46841 every 8 connection(s)
I20260809 06:19:34.563897 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:34.567919 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9736
I20260809 06:19:34.568138 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:34.568985 3621 external_mini_cluster.cc:1428] Setting key eac61f8b6892bc210bdeb1d99cfdca7a
Aug 09 06:19:34 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:34.584463 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:34.566419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48351 (local address 127.3.137.126:38645)
0809 06:19:34.566905 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:34.566910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:34.567894 (+ 984us) server_negotiation.cc:407] Connection header received
0809 06:19:34.570257 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:34.570262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:34.570339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:34.570469 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:34.572089 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:34.572806 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:34.574374 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:34.574784 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:34.574925 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:34.578532 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:34.578537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:34.578540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:34.578570 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:34.581252 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:34.581990 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:34.582000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:34.582002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:34.582120 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:34.582568 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:34.582573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:34.582576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:34.582850 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:34.583056 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:34.584032 (+ 976us) server_negotiation.cc:299] Negotiation successful
0809 06:19:34.584201 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":260,"threads_started":1}
I20260809 06:19:34.586046 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:34.566846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:38645 (local address 127.3.137.65:48351)
0809 06:19:34.567703 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:34.567751 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:34.569967 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:34.570737 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:34.570749 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:34.571387 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:34.571923 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:34.571941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:34.572978 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:34.572982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:34.574227 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:34.574239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:34.575043 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:34.575053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:34.575295 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:34.575976 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:34.575997 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:34.578317 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:34.581398 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:34.581408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:34.581423 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:34.581855 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:34.582234 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:34.582239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:34.582243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:34.582441 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:34.583101 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:34.583113 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:34.583561 (+ 448us) client_negotiation.cc:769] Sending connection context
0809 06:19:34.583917 (+ 356us) client_negotiation.cc:240] Negotiation successful
0809 06:19:34.584511 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":334,"threads_started":1}
I20260809 06:19:34.587823 9878 heartbeater.cc:347] Connected to a master server at 127.3.137.126:38645
I20260809 06:19:34.588153 9878 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:34.588884 9878 heartbeater.cc:511] Master 127.3.137.126:38645 requested a full tablet report, sending...
I20260809 06:19:34.591336 9670 ts_manager.cc:194] Registered new tserver with Master: c9643e9173f04da6ae93503cbe91d7d0 (127.3.137.65:46841)
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:19:34.593185 9670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:48351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:34.623813 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:34.964110 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:34.964368 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:34.964520 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:34.971603 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:34.971705 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:34.971920 9885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:34.981518 9885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:34.982729 9885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:34.984120 9885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:34.997077 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:34.997202 9893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:34.997249 9895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:34.998220 9885 server_base.cc:1034] running on GCE node
I20260809 06:19:34.998816 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:34.999758 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:35.000979 9885 hybrid_clock.cc:648] HybridClock initialized: now 1786256375000921 us; error 52 us; skew 500 ppm
I20260809 06:19:35.001288 9885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:34 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.004603 9885 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:35.006047 9885 webserver.cc:460] Webserver started at http://127.3.137.66:42491/ using document root <none> and password file <none>
I20260809 06:19:35.006878 9885 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:35.007041 9885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:35.007381 9885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:35.010723 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "976d64f837544816a7f1ce588a3b4263"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "6f01bddd99949ecf182374699b707a58"
server_key_iv: "709c40a2e38f0733a30ab949ed34c2ff"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.011657 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "976d64f837544816a7f1ce588a3b4263"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "6f01bddd99949ecf182374699b707a58"
server_key_iv: "709c40a2e38f0733a30ab949ed34c2ff"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.016875 9885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:35.020287 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.021526 9885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:35.021705 9885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "976d64f837544816a7f1ce588a3b4263"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "6f01bddd99949ecf182374699b707a58"
server_key_iv: "709c40a2e38f0733a30ab949ed34c2ff"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.021919 9885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:35.050608 9885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:35.054949 9885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:35.055312 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:35.056296 9885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:35.057648 9885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:35.057751 9885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.057909 9885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:35.057981 9885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.094225 9885 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40903
I20260809 06:19:35.094277 10027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40903 every 8 connection(s)
I20260809 06:19:35.095606 9885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:35.096530 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 9885
I20260809 06:19:35.096711 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:35.097339 3621 external_mini_cluster.cc:1428] Setting key 452b97f7b3beb4e532095e43b15a5072
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:35.117971 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.098652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:41881 (local address 127.3.137.126:38645)
0809 06:19:35.099210 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.099216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.100116 (+ 900us) server_negotiation.cc:407] Connection header received
0809 06:19:35.102846 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.102850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.102912 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.103073 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.105176 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.105937 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.107824 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.108365 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.108503 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.112243 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.112249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.112256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.112294 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.114913 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.115680 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.115685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.115689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.115832 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.116265 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.116270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.116273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.116529 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.116687 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.117349 (+ 662us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.117572 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":241,"threads_started":1}
I20260809 06:19:35.118918 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.099280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:38645 (local address 127.3.137.66:41881)
0809 06:19:35.099963 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:35.100004 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:35.102552 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:35.103321 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:35.103336 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:35.104121 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:35.104918 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:35.104946 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.106120 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.106125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:35.107677 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:35.107691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.108598 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.108605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:35.108831 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.109545 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:35.109567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:35.112026 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:35.115114 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:35.115123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:35.115138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:35.115551 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:35.115955 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:35.115960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:35.115964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:35.116143 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:35.116661 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:35.116668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:35.117005 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 06:19:35.117255 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:19:35.117788 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":789,"threads_started":1}
I20260809 06:19:35.120733 10028 heartbeater.cc:347] Connected to a master server at 127.3.137.126:38645
I20260809 06:19:35.121043 10028 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:35.121731 10028 heartbeater.cc:511] Master 127.3.137.126:38645 requested a full tablet report, sending...
I20260809 06:19:35.123347 9670 ts_manager.cc:194] Registered new tserver with Master: 976d64f837544816a7f1ce588a3b4263 (127.3.137.66:40903)
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:19:35.124600 9670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:41881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:35.153847 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:35.478189 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:35.478502 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:35.478683 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:35.486027 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:35.486186 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:35.486357 10034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:35.496392 10034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:35.497761 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:35.499238 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:35.512490 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:35.512529 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:35.512446 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:35.513902 10034 server_base.cc:1034] running on GCE node
I20260809 06:19:35.514744 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:35.516036 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:35.517300 10034 hybrid_clock.cc:648] HybridClock initialized: now 1786256375517247 us; error 61 us; skew 500 ppm
I20260809 06:19:35.517645 10034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.521209 10034 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:35.522651 10034 webserver.cc:460] Webserver started at http://127.3.137.67:40161/ using document root <none> and password file <none>
I20260809 06:19:35.523483 10034 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:35.523593 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:35.523865 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:35.527025 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "9549675f25f345858e7cd855253a62dc"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "c513a5e3ae00f080f3328a4f4cc3308b"
server_key_iv: "21a88255bf3230781de73a45ee709ca3"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.527837 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "9549675f25f345858e7cd855253a62dc"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "c513a5e3ae00f080f3328a4f4cc3308b"
server_key_iv: "21a88255bf3230781de73a45ee709ca3"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.533273 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 06:19:35.536903 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.538261 10034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:19:35.538451 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "9549675f25f345858e7cd855253a62dc"
format_stamp: "Formatted at 2026-08-09 06:19:35 on dist-test-slave-2w3w"
server_key: "c513a5e3ae00f080f3328a4f4cc3308b"
server_key_iv: "21a88255bf3230781de73a45ee709ca3"
server_key_version: "encryptionkey@0"
I20260809 06:19:35.538671 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:35.569695 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:35.573551 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:35.573882 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:35.574862 10034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:35.576171 10034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:35.576247 10034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.576419 10034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:35.576500 10034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:35.596508 9878 heartbeater.cc:503] Master 127.3.137.126:38645 was elected leader, sending a full tablet report...
I20260809 06:19:35.610646 10034 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:40603
I20260809 06:19:35.610679 10171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:40603 every 8 connection(s)
I20260809 06:19:35.612076 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:35.615620 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10034
I20260809 06:19:35.615806 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:35.616313 3621 external_mini_cluster.cc:1428] Setting key ef398fc9842adaaad918a06566e91aa1
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:35.632797 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.614542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47353 (local address 127.3.137.126:38645)
0809 06:19:35.614725 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.614730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.616115 (+ 1385us) server_negotiation.cc:407] Connection header received
0809 06:19:35.618223 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.618229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.618336 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.618439 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.620501 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.621240 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.622902 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.623333 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.623428 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.626778 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.626783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.626786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.626818 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.629507 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.630181 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.630190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.630193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.630336 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.630994 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.631001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.631005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.631341 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.631499 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.632347 (+ 848us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.632520 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:19:35.633692 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.615131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:38645 (local address 127.3.137.67:47353)
0809 06:19:35.615942 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:35.615994 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:35.617908 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:35.618807 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:35.618819 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:35.619722 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:35.620337 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:35.620356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.621390 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.621394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:35.622769 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:35.622784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.623528 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.623534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:35.623738 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.624343 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:35.624364 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:35.626585 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:35.629640 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:35.629646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:35.629658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:35.630059 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:35.630550 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:35.630558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:35.630563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:35.630825 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:35.631462 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:35.631468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:35.631943 (+ 475us) client_negotiation.cc:769] Sending connection context
0809 06:19:35.632238 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 06:19:35.632629 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":371,"threads_started":1}
I20260809 06:19:35.635149 10172 heartbeater.cc:347] Connected to a master server at 127.3.137.126:38645
I20260809 06:19:35.635401 10172 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:35.636078 10172 heartbeater.cc:511] Master 127.3.137.126:38645 requested a full tablet report, sending...
I20260809 06:19:35.637447 9670 ts_manager.cc:194] Registered new tserver with Master: 9549675f25f345858e7cd855253a62dc (127.3.137.67:40603)
I20260809 06:19:35.638172 9670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:47353
I20260809 06:19:35.644171 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/test-admin.keytab.
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.673193 3621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 06:19:35.678586 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:35.676128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.3.137.65:46841)
0809 06:19:35.676797 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.676804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.676832 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:19:35.676894 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.676900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.677113 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.677242 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.678002 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33436: BlockingRecv error: recv got EOF from 127.0.0.1:33436 (error 108)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":350,"threads_started":1}
W20260809 06:19:35.678648 3621 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc' of type FILE
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.680341 3621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.681746 3621 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:35.697202 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.683447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.3.137.65:46841)
0809 06:19:35.683626 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.683630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.683711 (+ 81us) server_negotiation.cc:407] Connection header received
0809 06:19:35.683895 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.683899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.683960 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.684051 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.685061 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.686069 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.687337 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.687883 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.687998 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.691575 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.691585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.691600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.691639 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.694481 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.695204 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.695211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.695214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.695333 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.695651 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.695654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.695657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.696000 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.696217 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.696643 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.696920 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 06:19:35.703109 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:35.701493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.3.137.65:46841)
0809 06:19:35.701698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.701703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.701764 (+ 61us) server_negotiation.cc:407] Connection header received
0809 06:19:35.701911 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.701915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.701964 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.702037 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.702840 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33442: BlockingRecv error: recv got EOF from 127.0.0.1:33442 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 06:19:35.703339 3621 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc' of type FILE
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.704802 3621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.706103 3621 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:35.720067 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.707541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.3.137.65:46841)
0809 06:19:35.707750 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.707755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.707976 (+ 221us) server_negotiation.cc:407] Connection header received
0809 06:19:35.708115 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.708120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.708182 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.708256 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.709183 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.709989 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.711289 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.711700 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.711822 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.715121 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.715126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.715129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.715157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.717755 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.718312 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.718315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.718318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.718429 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.718753 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.718756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.718758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.719091 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.719274 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.719584 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.719831 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 06:19:35.724004 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:35.722429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.3.137.65:46841)
0809 06:19:35.722613 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.722619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.722801 (+ 182us) server_negotiation.cc:407] Connection header received
0809 06:19:35.722949 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.722998 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.723075 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.723151 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.723791 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33454: BlockingRecv error: recv got EOF from 127.0.0.1:33454 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 06:19:35.724160 3621 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786256324325922-3621-0/krb5kdc/krb5cc' of type FILE
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.725771 3621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:35.727114 3621 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:35.741716 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.728311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.3.137.65:46841)
0809 06:19:35.728507 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.728514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.728773 (+ 259us) server_negotiation.cc:407] Connection header received
0809 06:19:35.728944 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.728949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.729021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.729120 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.730073 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.731193 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.732767 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.733519 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.733618 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.736928 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.736933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.736936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.736970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.739576 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.740131 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.740134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.740136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.740274 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.740576 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.740579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.740581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.740814 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.741043 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.741210 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.741361 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:19:35.743352 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9736
I20260809 06:19:35.754374 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.744057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.3.137.65:46841)
0809 06:19:35.744230 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.744238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.744512 (+ 274us) server_negotiation.cc:407] Connection header received
0809 06:19:35.744603 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.744606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.744654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.744726 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.745606 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.746231 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.747416 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.747814 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.747905 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.749675 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.749680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.749682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.749709 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.752117 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.752635 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.752638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.752641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.752765 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.753177 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.753180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.753182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.753396 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.753584 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.753865 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.754021 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 06:19:35.755038 9873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:35.838935 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9736
I20260809 06:19:35.856683 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9885
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:35.871830 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.857588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.3.137.66:40903)
0809 06:19:35.858196 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.858203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.858230 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:19:35.858291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.858297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.858513 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.858642 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.859656 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.860709 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.861914 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.862458 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.862554 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.866385 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.866396 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.866410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.866441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.869301 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.869986 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.869992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.869995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.870119 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.870444 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.870447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.870449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.870770 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.871074 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.871353 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.871552 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":336,"threads_started":1}
I20260809 06:19:35.873116 10023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:35.955669 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9885
I20260809 06:19:35.974045 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10034
Aug 09 06:19:35 dist-test-slave-2w3w krb5kdc[9622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:35.990229 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:35.975037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.3.137.67:40603)
0809 06:19:35.975634 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:35.975640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:35.975670 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:19:35.975738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:35.975745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:35.975958 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:35.976129 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:35.977311 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.978363 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.979776 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:35.980347 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:35.980500 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:35.984451 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:35.984463 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:35.984478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:35.984512 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:35.987607 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.988272 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.988279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.988281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.988399 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:35.988729 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:35.988732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:35.988734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:35.989074 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:35.989378 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:35.989651 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 06:19:35.989925 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":278,"threads_started":1}
I20260809 06:19:35.991576 10167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:36.074434 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10034
I20260809 06:19:36.092268 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 9636
I20260809 06:19:36.093432 9701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:36.196856 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 9636
2026-08-09T06:19:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4034 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:36 dist-test-slave-2w3w krb5kdc[10203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:36 dist-test-slave-2w3w krb5kdc[10203](info): set up 2 sockets
Aug 09 06:19:36 dist-test-slave-2w3w krb5kdc[10203](info): commencing operation
krb5kdc: starting...
W20260809 06:19:37.348307 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:37 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab.
Entry for principal oryx/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab.
I20260809 06:19:37.568351 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45079
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab
--principal=oryx/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:37.892261 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:37.892529 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:37.892622 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:37.899878 10217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:37.899984 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:37.900030 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:37.900080 10217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:37.900125 10217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:37.909529 10217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab
--principal=oryx/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:37.910722 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:37.911959 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:37.923416 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:37.923375 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:37.923473 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:37.924990 10217 server_base.cc:1034] running on GCE node
I20260809 06:19:37.925738 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:37.927199 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:37.928436 10217 hybrid_clock.cc:648] HybridClock initialized: now 1786256377928379 us; error 42 us; skew 500 ppm
I20260809 06:19:37.928771 10217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:37 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:37.932442 10217 init.cc:377] Logged in from keytab as oryx/127.3.137.126@KRBTEST.COM (short username oryx)
I20260809 06:19:37.934001 10217 webserver.cc:460] Webserver started at http://127.3.137.126:43833/ using document root <none> and password file <none>
I20260809 06:19:37.934701 10217 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:37.934872 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:37.935226 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:37.938164 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "efb59627dc9d4d75ab7f052e5bdbf070"
format_stamp: "Formatted at 2026-08-09 06:19:37 on dist-test-slave-2w3w"
server_key: "c89cdd8f6a653de64fab3bd47a1bac6b"
server_key_iv: "e19058be236a37938166677975e5d6bf"
server_key_version: "encryptionkey@0"
I20260809 06:19:37.938938 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "efb59627dc9d4d75ab7f052e5bdbf070"
format_stamp: "Formatted at 2026-08-09 06:19:37 on dist-test-slave-2w3w"
server_key: "c89cdd8f6a653de64fab3bd47a1bac6b"
server_key_iv: "e19058be236a37938166677975e5d6bf"
server_key_version: "encryptionkey@0"
I20260809 06:19:37.944135 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 06:19:37.947474 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:37.948725 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:19:37.948931 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "efb59627dc9d4d75ab7f052e5bdbf070"
format_stamp: "Formatted at 2026-08-09 06:19:37 on dist-test-slave-2w3w"
server_key: "c89cdd8f6a653de64fab3bd47a1bac6b"
server_key_iv: "e19058be236a37938166677975e5d6bf"
server_key_version: "encryptionkey@0"
I20260809 06:19:37.949134 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:37.972229 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:37.976541 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:37.976836 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:37.995537 10217 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45079
I20260809 06:19:37.995589 10288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45079 every 8 connection(s)
I20260809 06:19:37.996940 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:37.997684 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10217
I20260809 06:19:37.997929 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:37.998744 3621 external_mini_cluster.cc:1428] Setting key e2b6f7a5404f17cc658111fe50318641
I20260809 06:19:38.001379 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:19:38 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:38.014833 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Bootstrap starting.
I20260809 06:19:38.019132 10289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:38.020383 10289 log.cc:824] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:38.023597 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:38.000880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.3.137.126:45079)
0809 06:19:38.001706 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:38.001725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:38.001767 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:19:38.003428 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:38.003449 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:38.003823 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:38.004148 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:38.005972 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.007364 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.009179 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.009849 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.010002 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:38.014766 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:38.014777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:38.014795 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:38.014853 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:38.018851 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:38.019650 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:38.019658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:38.019666 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:38.019867 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:38.020236 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:38.020240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:38.020244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:38.020993 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:38.021298 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:38.021797 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 06:19:38.022256 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260809 06:19:38.024015 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: No bootstrap required, opened a new log
I20260809 06:19:38.031538 10289 raft_consensus.cc:348] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } }
I20260809 06:19:38.031881 10289 raft_consensus.cc:374] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:38.031999 10289 raft_consensus.cc:729] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb59627dc9d4d75ab7f052e5bdbf070, State: Initialized, Role: FOLLOWER
I20260809 06:19:38.032642 10289 consensus_queue.cc:260] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } }
I20260809 06:19:38.032902 10289 raft_consensus.cc:388] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:38.033088 10289 raft_consensus.cc:482] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:38.033286 10289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:38.035530 10289 raft_consensus.cc:504] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } }
I20260809 06:19:38.036047 10289 leader_election.cc:302] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efb59627dc9d4d75ab7f052e5bdbf070; no voters:
I20260809 06:19:38.036777 10289 leader_election.cc:288] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:38.036896 10295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:38.037674 10295 raft_consensus.cc:686] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 LEADER]: Becoming Leader. State: Replica: efb59627dc9d4d75ab7f052e5bdbf070, State: Running, Role: LEADER
I20260809 06:19:38.038213 10295 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } }
I20260809 06:19:38.039006 10289 sys_catalog.cc:564] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:38.040928 10297 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb59627dc9d4d75ab7f052e5bdbf070. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } }
I20260809 06:19:38.040939 10296 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } }
I20260809 06:19:38.041342 10297 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:38.041342 10296 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:38.043049 10301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:38.047283 10301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:38.056209 10301 catalog_manager.cc:1347] Generated new cluster ID: a0b918c1c940455382b3d40324d9b228
I20260809 06:19:38.056325 10301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:38.067116 10301 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:38.068365 10301 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:38.074878 10301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Generated new TSK 0
I20260809 06:19:38.075785 10301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
I20260809 06:19:38.146428 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:38.470031 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:38.470292 10317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:38.470485 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:38.477490 10317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:38.477581 10317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:38.477728 10317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:38.487018 10317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:38.488219 10317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:38.489521 10317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:38.502720 10325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:38.502676 10323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:38.502765 10322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:38.508460 10317 server_base.cc:1034] running on GCE node
I20260809 06:19:38.509096 10317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:38.510015 10317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:38.511237 10317 hybrid_clock.cc:648] HybridClock initialized: now 1786256378511202 us; error 46 us; skew 500 ppm
I20260809 06:19:38.511544 10317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:38 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256378, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:38.514679 10317 init.cc:377] Logged in from keytab as oryx/127.3.137.65@KRBTEST.COM (short username oryx)
I20260809 06:19:38.516232 10317 webserver.cc:460] Webserver started at http://127.3.137.65:40881/ using document root <none> and password file <none>
I20260809 06:19:38.516945 10317 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:38.517091 10317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:38.517390 10317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:38.520414 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "589fed3fea814338b69691e8e33a08d7"
format_stamp: "Formatted at 2026-08-09 06:19:38 on dist-test-slave-2w3w"
server_key: "c544717a07849735b6f3924cab7b8d63"
server_key_iv: "6dfb52009191db6e57898452f26500d8"
server_key_version: "encryptionkey@0"
I20260809 06:19:38.521189 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "589fed3fea814338b69691e8e33a08d7"
format_stamp: "Formatted at 2026-08-09 06:19:38 on dist-test-slave-2w3w"
server_key: "c544717a07849735b6f3924cab7b8d63"
server_key_iv: "6dfb52009191db6e57898452f26500d8"
server_key_version: "encryptionkey@0"
I20260809 06:19:38.526551 10317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:19:38.529925 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:38.531399 10317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:38.531581 10317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "589fed3fea814338b69691e8e33a08d7"
format_stamp: "Formatted at 2026-08-09 06:19:38 on dist-test-slave-2w3w"
server_key: "c544717a07849735b6f3924cab7b8d63"
server_key_iv: "6dfb52009191db6e57898452f26500d8"
server_key_version: "encryptionkey@0"
I20260809 06:19:38.531828 10317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:38.547472 10317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:38.551993 10317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:38.552289 10317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:38.553300 10317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:38.554579 10317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:38.554692 10317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:38.554862 10317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:38.554948 10317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:38.589133 10317 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40567
I20260809 06:19:38.589195 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40567 every 8 connection(s)
I20260809 06:19:38.590557 10317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:38.600881 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10317
I20260809 06:19:38.601140 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:38.601900 3621 external_mini_cluster.cc:1428] Setting key ef6e5b502daebd1f9cd9b8668151a749
Aug 09 06:19:38 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256378, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:38.612130 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:38.593268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48277 (local address 127.3.137.126:45079)
0809 06:19:38.593778 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:38.593785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:38.594857 (+ 1072us) server_negotiation.cc:407] Connection header received
0809 06:19:38.596650 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:38.596654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:38.596725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:38.596839 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:38.598281 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.599149 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.601008 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.601528 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.601639 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:38.606109 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:38.606114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:38.606117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:38.606149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:38.608770 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:38.609749 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:38.609753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:38.609756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:38.609869 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:38.610384 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:38.610387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:38.610388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:38.610641 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:38.610765 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:38.611708 (+ 943us) server_negotiation.cc:299] Negotiation successful
0809 06:19:38.611872 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260809 06:19:38.613562 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:38.593865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:48277)
0809 06:19:38.594661 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:38.594735 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:38.596371 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:38.597036 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:38.597044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:38.597558 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:38.598100 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:38.598117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.599302 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.599306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:38.600809 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:38.600827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:38.601774 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:38.601783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:38.602127 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:38.602874 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:38.602896 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:38.605875 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:38.608913 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:38.608926 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:38.608948 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:38.609637 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:38.609987 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:38.609995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:38.610002 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:38.610273 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:38.610779 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:38.610790 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:38.611297 (+ 507us) client_negotiation.cc:769] Sending connection context
0809 06:19:38.611635 (+ 338us) client_negotiation.cc:240] Negotiation successful
0809 06:19:38.612256 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":288,"threads_started":1}
I20260809 06:19:38.615224 10459 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:38.615553 10459 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:38.616400 10459 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
I20260809 06:19:38.618623 10254 ts_manager.cc:194] Registered new tserver with Master: 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:38.620142 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:48277
WARNING: no policy specified for oryx/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
I20260809 06:19:38.658128 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:39.001032 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:39.001374 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:39.001569 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:39.009799 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:39.009954 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:39.010160 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:39.021381 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:39.022733 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:39.024204 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:39.037998 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:39.037971 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:39.038045 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:39.039217 10465 server_base.cc:1034] running on GCE node
I20260809 06:19:39.039873 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:39.040865 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:39.042057 10465 hybrid_clock.cc:648] HybridClock initialized: now 1786256379041995 us; error 53 us; skew 500 ppm
I20260809 06:19:39.042373 10465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:39 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256379, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:39.046111 10465 init.cc:377] Logged in from keytab as oryx/127.3.137.66@KRBTEST.COM (short username oryx)
I20260809 06:19:39.048640 10465 webserver.cc:460] Webserver started at http://127.3.137.66:44613/ using document root <none> and password file <none>
I20260809 06:19:39.049530 10465 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:39.049701 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:39.050061 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:39.053499 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "efba900d3fac4e4dbf5f89860749f96c"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "6a07d4bace350c78567bb345711c038c"
server_key_iv: "b6614c653d36201a57f0a8c049a9e145"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.054268 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "efba900d3fac4e4dbf5f89860749f96c"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "6a07d4bace350c78567bb345711c038c"
server_key_iv: "b6614c653d36201a57f0a8c049a9e145"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.059520 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:19:39.062872 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:39.064188 10465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:39.064345 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "efba900d3fac4e4dbf5f89860749f96c"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "6a07d4bace350c78567bb345711c038c"
server_key_iv: "b6614c653d36201a57f0a8c049a9e145"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.064548 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:39.088038 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:39.092187 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:39.092520 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:39.093461 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:39.094802 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:39.094883 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:39.095021 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:39.095073 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 06:19:39.128199 10465 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:37993
I20260809 06:19:39.128301 10602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:37993 every 8 connection(s)
I20260809 06:19:39.129503 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:39.130405 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10465
I20260809 06:19:39.130612 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:39.131244 3621 external_mini_cluster.cc:1428] Setting key 402dfe90e41f26527c51996f5b3629a6
Aug 09 06:19:39 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256379, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:39.155579 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:39.132802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:36853 (local address 127.3.137.126:45079)
0809 06:19:39.133379 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:39.133386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:39.134579 (+ 1193us) server_negotiation.cc:407] Connection header received
0809 06:19:39.137851 (+ 3272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:39.137855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:39.137919 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:39.138020 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:39.140404 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.141481 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.144176 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.144932 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.145071 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:39.148865 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:39.148871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:39.148874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:39.148909 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:39.151964 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:39.152932 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:39.152936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:39.152939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:39.153074 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:39.153616 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:39.153619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:39.153622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:39.153927 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:39.154099 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:39.154856 (+ 757us) server_negotiation.cc:299] Negotiation successful
0809 06:19:39.155149 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":284,"threads_started":1}
I20260809 06:19:39.156618 10606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:39.133445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:36853)
0809 06:19:39.134391 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:39.134441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:39.137501 (+ 3060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:39.138324 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:39.138339 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:39.139292 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:39.140191 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:39.140221 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.141659 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.141668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:39.144025 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:39.144038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.145156 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.145162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:39.145382 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:39.146127 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:39.146151 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:39.148607 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:39.152115 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:39.152125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:39.152144 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:39.152783 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:39.153200 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:39.153205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:39.153209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:39.153464 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:39.154078 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:39.154087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:39.154457 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 06:19:39.154756 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 06:19:39.155384 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for oryx/127.3.137.67@KRBTEST.COM; defaulting to no policy
I20260809 06:19:39.158308 10603 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:39.158654 10603 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:39.159449 10603 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.67@KRBTEST.COM" created.
I20260809 06:19:39.161221 10254 ts_manager.cc:194] Registered new tserver with Master: efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
I20260809 06:19:39.162539 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:36853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
I20260809 06:19:39.191828 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:39.515913 10609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:39.516187 10609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:39.516281 10609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:39.523581 10609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:39.523684 10609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:39.523823 10609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:39.533408 10609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:39.534592 10609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:39.535941 10609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:39.549054 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:39.549127 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:39.549152 10615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:39.550037 10609 server_base.cc:1034] running on GCE node
I20260809 06:19:39.550644 10609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:39.551720 10609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:39.552948 10609 hybrid_clock.cc:648] HybridClock initialized: now 1786256379552910 us; error 49 us; skew 500 ppm
I20260809 06:19:39.553301 10609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:39 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256379, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:39.556661 10609 init.cc:377] Logged in from keytab as oryx/127.3.137.67@KRBTEST.COM (short username oryx)
I20260809 06:19:39.558038 10609 webserver.cc:460] Webserver started at http://127.3.137.67:39773/ using document root <none> and password file <none>
I20260809 06:19:39.558722 10609 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:39.558873 10609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:39.559283 10609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:39.562156 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "aab95a02b3d51bcf028dc8d4d4fe9328"
server_key_iv: "cc8df043913ddf08e2471160ac3b1109"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.562924 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "aab95a02b3d51bcf028dc8d4d4fe9328"
server_key_iv: "cc8df043913ddf08e2471160ac3b1109"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.568178 10609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:19:39.571719 10627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:39.572990 10609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:39.573163 10609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "aab95a02b3d51bcf028dc8d4d4fe9328"
server_key_iv: "cc8df043913ddf08e2471160ac3b1109"
server_key_version: "encryptionkey@0"
I20260809 06:19:39.573386 10609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:39.595256 10609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:39.599714 10609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:39.600029 10609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:39.600983 10609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:39.602303 10609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:39.602411 10609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:39.602557 10609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:39.602630 10609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:39.623870 10459 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
I20260809 06:19:39.637058 10609 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:45815
I20260809 06:19:39.637153 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:45815 every 8 connection(s)
I20260809 06:19:39.638353 10609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:39.645157 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10609
I20260809 06:19:39.645331 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:39.645872 3621 external_mini_cluster.cc:1428] Setting key 8093702899ff31e528a7e2fefed4b902
Aug 09 06:19:39 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256379, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:39.661084 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:39.640998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:38245 (local address 127.3.137.126:45079)
0809 06:19:39.641208 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:39.641214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:39.642431 (+ 1217us) server_negotiation.cc:407] Connection header received
0809 06:19:39.644238 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:39.644243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:39.644308 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:39.644394 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:39.645901 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.646995 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.648734 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.649099 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.649189 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:39.655252 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:39.655256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:39.655259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:39.655298 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:39.657889 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:39.658569 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:39.658573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:39.658575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:39.658693 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:39.659129 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:39.659134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:39.659137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:39.659507 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:39.659676 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:39.660577 (+ 901us) server_negotiation.cc:299] Negotiation successful
0809 06:19:39.660794 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:39.662166 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:39.641580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:38245)
0809 06:19:39.642254 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:39.642299 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:39.643996 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:39.644588 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:39.644595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:39.645063 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:39.645725 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:39.645742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.647160 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.647164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:39.648588 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:39.648601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:39.649315 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:39.649321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:39.649527 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:39.652731 (+ 3204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:39.652753 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:39.655077 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:39.658007 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:39.658013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:39.658024 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:39.658440 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:39.658784 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:39.658787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:39.658790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:39.658953 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:39.659655 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:39.659660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:39.660135 (+ 475us) client_negotiation.cc:769] Sending connection context
0809 06:19:39.660414 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 06:19:39.660911 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260809 06:19:39.663587 10754 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:39.663889 10754 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:39.664515 10754 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
I20260809 06:19:39.665825 10254 ts_manager.cc:194] Registered new tserver with Master: 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815)
I20260809 06:19:39.666548 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:38245
I20260809 06:19:39.673295 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.137.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.125.keytab.
Entry for principal oryx/127.3.137.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.125.keytab.
I20260809 06:19:39.727182 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:38263
--webserver_interface=127.3.137.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.125.keytab
--principal=oryx/127.3.137.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45079,127.3.137.125:38263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:40.069985 10760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:40.070375 10760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:40.070544 10760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:40.078317 10760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:40.078445 10760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:40.078548 10760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:40.078653 10760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:40.078739 10760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:40.089351 10760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45079,127.3.137.125:38263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.125.keytab
--principal=oryx/127.3.137.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.137.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
W20260809 06:19:40.089811 10760 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.137.126:45079,127.3.137.125:38263'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 06:19:40.091329 10760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:40.093165 10760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:40.105821 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:40.105849 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:40.106375 10760 server_base.cc:1034] running on GCE node
W20260809 06:19:40.105897 10765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:40.107342 10760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:40.108748 10760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:40.110009 10760 hybrid_clock.cc:648] HybridClock initialized: now 1786256380109943 us; error 40 us; skew 500 ppm
I20260809 06:19:40.110338 10760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:40.113503 10760 init.cc:377] Logged in from keytab as oryx/127.3.137.125@KRBTEST.COM (short username oryx)
I20260809 06:19:40.115146 10760 webserver.cc:460] Webserver started at http://127.3.137.125:43621/ using document root <none> and password file <none>
I20260809 06:19:40.115833 10760 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:40.115969 10760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:40.116294 10760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:40.119441 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/instance:
uuid: "7105adb3d3484f50ba33155705b6b3e8"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "53b44ab2ee0b287eae8c19769dfc5791"
server_key_iv: "620e6bb93bee6d49b62d56133b101aec"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.120297 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal/instance:
uuid: "7105adb3d3484f50ba33155705b6b3e8"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "53b44ab2ee0b287eae8c19769dfc5791"
server_key_iv: "620e6bb93bee6d49b62d56133b101aec"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.125396 10760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:19:40.128944 10775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:40.130349 10760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:19:40.130518 10760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "7105adb3d3484f50ba33155705b6b3e8"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "53b44ab2ee0b287eae8c19769dfc5791"
server_key_iv: "620e6bb93bee6d49b62d56133b101aec"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.130717 10760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:40.155943 10760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:40.159983 10760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:40.160264 10760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:40.165180 10603 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
I20260809 06:19:40.179978 10760 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.125:38263
I20260809 06:19:40.180002 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.125:38263 every 8 connection(s)
I20260809 06:19:40.181471 10760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
I20260809 06:19:40.185155 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:40.189939 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10760
I20260809 06:19:40.192783 10831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.125@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:40.213443 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.195403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.3.137.126:45079)
0809 06:19:40.196881 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.196886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.196901 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:19:40.196957 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.196961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.197015 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.197100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.198730 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.199871 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.202061 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.202683 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.202831 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.206998 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.207005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.207016 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.207048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.209935 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.211053 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.211057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.211059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.211181 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.211666 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.211672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.211675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.211977 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.212141 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.212820 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.213046 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1275,"threads_started":1}
I20260809 06:19:40.214493 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.195358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45100)
0809 06:19:40.196141 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.196196 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.196397 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.197211 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.197220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.198063 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.198557 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.198572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.200019 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.200022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.201917 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.201935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.202950 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.202993 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.203206 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.203934 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.203961 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.206743 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.210077 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.210088 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.210100 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.210880 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.211312 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.211318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.211324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.211548 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.212126 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.212133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.212460 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.212741 (+ 281us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.213232 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":3,"thread_start_us":302,"threads_started":1}
I20260809 06:19:40.214524 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.192397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.3.137.125:38263)
0809 06:19:40.193216 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.193230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.193277 (+ 47us) server_negotiation.cc:407] Connection header received
0809 06:19:40.195249 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.195275 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.195830 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.196198 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.198183 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.200135 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.201924 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.202725 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.202878 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.207776 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.207784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.207796 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.207828 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.210983 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.211642 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.211650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.211653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.211782 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.212182 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.212188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.212192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.212542 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.212864 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.213194 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.213443 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":625,"thread_start_us":403,"threads_started":1}
I20260809 06:19:40.216585 10831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.125@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:40.228557 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.217525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46822)
0809 06:19:40.217702 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.217720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.217822 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.218173 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.218178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.218500 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.218623 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.218631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.219609 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.219612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.220462 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.220473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.221095 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.221099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.221227 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.221700 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.221716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.223700 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.226554 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.226558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.226560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.226897 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.227269 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.227272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.227275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.227403 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.227842 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.227845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.227960 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.228067 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.228271 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 06:19:40.228667 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.217513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.3.137.125:38263)
0809 06:19:40.217700 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.217705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.217793 (+ 88us) server_negotiation.cc:407] Connection header received
0809 06:19:40.217946 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.217952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.218021 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.218094 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.218747 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.219483 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.220587 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.220972 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.221071 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.223882 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.223886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.223889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.223916 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.226442 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.227033 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.227037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.227039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.227174 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.227501 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.227505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.227506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.227745 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.227854 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.228146 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.228400 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":82}
I20260809 06:19:40.233336 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Bootstrap starting.
I20260809 06:19:40.236670 10831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:40.237670 10831 log.cc:824] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:40.240517 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: No bootstrap required, opened a new log
I20260809 06:19:40.247440 10831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } }
I20260809 06:19:40.247763 10831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:40.247891 10831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7105adb3d3484f50ba33155705b6b3e8, State: Initialized, Role: FOLLOWER
I20260809 06:19:40.248445 10831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } }
I20260809 06:19:40.249292 10839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } }
I20260809 06:19:40.249934 10839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:40.250012 10840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:40.250315 10840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } }
I20260809 06:19:40.250732 10831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:40.251883 10840 leader_election.cc:288] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079)
I20260809 06:19:40.252786 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7105adb3d3484f50ba33155705b6b3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" is_pre_election: true
I20260809 06:19:40.254129 10779 leader_election.cc:302] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7105adb3d3484f50ba33155705b6b3e8; no voters: efb59627dc9d4d75ab7f052e5bdbf070
I20260809 06:19:40.254565 10840 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 06:19:40.260861 10852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:19:40.261003 10852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:19:40.262905 10760 master_runner.cc:428] Detected that this master 7105adb3d3484f50ba33155705b6b3e8 is joining an existing cluster
I20260809 06:19:40.263016 10760 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.137.125:38263
I20260809 06:19:40.285485 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.267124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45104)
0809 06:19:40.268183 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.268215 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.268436 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.269018 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.269022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.269838 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.270417 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.270430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.271912 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.271916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.273845 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.273860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.275113 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.275124 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.275402 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.276190 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.276223 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.277820 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.281980 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.281986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.281992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.282714 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.283235 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.283241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.283245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.283421 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.284178 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.284183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.284396 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.284621 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.284961 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":269,"spinlock_wait_cycles":33024,"thread_start_us":540,"threads_started":1}
I20260809 06:19:40.285992 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.267779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.3.137.125:38263)
0809 06:19:40.267964 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.267974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.268662 (+ 688us) server_negotiation.cc:407] Connection header received
0809 06:19:40.268843 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.268847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.268905 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.268966 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.270562 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.271694 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.273988 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.274703 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.274800 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.277991 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.277999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.278003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.278035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.282767 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.283635 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.283642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.283649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.283804 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.284219 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.284226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.284232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.284647 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.284776 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.285285 (+ 509us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.285543 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:19:40.286398 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.266971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.3.137.126:45079)
0809 06:19:40.267147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.267151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.268324 (+ 1173us) server_negotiation.cc:407] Connection header received
0809 06:19:40.268607 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.268614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.268733 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.268817 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.270584 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.271694 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.274004 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.274660 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.274787 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.278014 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.278021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.278025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.278063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.281825 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.282873 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.282879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.282883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.283089 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.283567 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.283572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.283575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.283965 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.284104 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.285799 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.286070 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 06:19:40.287547 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.267146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46824)
0809 06:19:40.268583 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.268604 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.268730 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.269156 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.269161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.269838 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.270431 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.270440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.271910 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.271915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.273845 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.273859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.275141 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.275146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.275402 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.276343 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.276361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.277820 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.282917 (+ 5097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.282923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.282928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.283505 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.283943 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.283949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.283956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.284118 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.284856 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.284863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.285025 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.286212 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.287083 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":283,"spinlock_wait_cycles":6784,"thread_start_us":1038,"threads_started":1}
WARNING: no policy specified for oryx/127.3.137.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.124@KRBTEST.COM" created.
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.126@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:40.311421 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.293226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.3.137.125:38263)
0809 06:19:40.293392 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.293397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.294079 (+ 682us) server_negotiation.cc:407] Connection header received
0809 06:19:40.294481 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.294488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.294606 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.294729 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.296235 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.297481 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.299449 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.300276 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.300377 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.304630 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.304637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.304642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.304686 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.308430 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.309374 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.309378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.309380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.309501 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.309888 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.309893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.309896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.310154 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.310317 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.310782 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.311016 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:19:40.311466 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.293216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46832)
0809 06:19:40.293933 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.293996 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.294215 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.294793 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.294800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.295694 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.296055 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.296069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.297634 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.297640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.299288 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.299303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.300444 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.300451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.300669 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.301364 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.301389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.304425 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.308572 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.308587 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.308597 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.309202 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.309616 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.309623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.309626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.309764 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.310288 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.310296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.310471 (+ 175us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.310666 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.311016 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":310,"threads_started":1}
I20260809 06:19:40.313740 10254 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.137.125:38263: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8"
member_type: NON_VOTER
last_known_addr {
host: "127.3.137.125"
port: 38263
}
attrs {
promote: true
}
}
dest_uuid: "efb59627dc9d4d75ab7f052e5bdbf070"
cas_config_opid_index: -1
I20260809 06:19:40.314990 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } }
I20260809 06:19:40.325778 10803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:40.328498 10866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:40.329613 10803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 06:19:40.331239 10869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 06:19:40.333449 10866 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.137.125:38263
I20260809 06:19:40.334409 10760 master.cc:537] Master@127.3.137.125:38263 shutting down...
I20260809 06:19:40.334055 10868 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:40.335250 10868 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
W20260809 06:19:40.336068 10237 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 06:19:40.337900 10237 consensus_peers.cc:487] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 -> Peer 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125:38263): Couldn't send request to peer 7105adb3d3484f50ba33155705b6b3e8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.124.keytab.
Entry for principal oryx/127.3.137.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.124.keytab.
I20260809 06:19:40.340565 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:45459
--webserver_interface=127.3.137.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.124.keytab
--principal=oryx/127.3.137.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:19:40.351413 10760 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:40.351845 10760 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:40.352089 10760 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: stopping tablet replica
I20260809 06:19:40.359611 10760 master.cc:552] Master@127.3.137.125:38263 shutdown complete.
I20260809 06:19:40.359792 10760 master_runner.cc:305] Clearing existing system tablet
I20260809 06:19:40.360383 10760 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:40.360532 10760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:40.367751 10760 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 06:19:40.371330 10874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:40.371729 10760 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:40.372241 10760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:19:40.372386 10760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "7105adb3d3484f50ba33155705b6b3e8"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "53b44ab2ee0b287eae8c19769dfc5791"
server_key_iv: "620e6bb93bee6d49b62d56133b101aec"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.377723 10760 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:40.384821 10760 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 06:19:40.385084 10760 log.cc:1196] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 06:19:40.385615 10760 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Deleting consensus metadata
I20260809 06:19:40.385941 10760 master_runner.cc:315] Copying system tablet from 127.3.137.126:45079
I20260809 06:19:40.388574 10760 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: tablet copy: Beginning tablet copy session from remote peer at address 127.3.137.126:45079
I20260809 06:19:40.410554 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.389617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.3.137.126:45079)
0809 06:19:40.389802 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.389808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.395544 (+ 5736us) server_negotiation.cc:407] Connection header received
0809 06:19:40.395742 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.395748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.395843 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.395957 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.397009 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.398221 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.400147 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.400844 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.400975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.403519 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.403525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.403531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.403570 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.407166 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.408138 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.408144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.408148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.408305 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.408823 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.408828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.408832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.409191 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.409347 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.409946 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.410172 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:19:40.411204 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.389576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45120)
0809 06:19:40.395392 (+ 5816us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.395413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.395579 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.396021 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.396026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.396524 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.396829 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.396842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.398375 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.398383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.399973 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.399989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.401041 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.401048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.401230 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.402012 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.402043 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.403292 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.407314 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.407322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.407328 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.407982 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.408444 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.408451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.408455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.408674 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.409346 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.409353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.409569 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.409815 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.410268 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5670,"thread_start_us":5545,"threads_started":1}
I20260809 06:19:40.412156 10274 tablet_copy_service.cc:140] P efb59627dc9d4d75ab7f052e5bdbf070: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7105adb3d3484f50ba33155705b6b3e8 ({username='oryx', principal='oryx/127.3.137.125@KRBTEST.COM'} at 127.0.0.1:45120)
I20260809 06:19:40.412485 10274 tablet_copy_service.cc:161] P efb59627dc9d4d75ab7f052e5bdbf070: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7105adb3d3484f50ba33155705b6b3e8 at {username='oryx', principal='oryx/127.3.137.125@KRBTEST.COM'} at 127.0.0.1:45120: session id = 7105adb3d3484f50ba33155705b6b3e8-00000000000000000000000000000000
I20260809 06:19:40.417129 10274 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 06:19:40.419385 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 06:19:40.426306 10760 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: tablet copy: Starting download of 0 data blocks...
I20260809 06:19:40.426687 10760 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: tablet copy: Starting download of 1 WAL segments...
I20260809 06:19:40.430032 10760 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 06:19:40.434532 10274 tablet_copy_service.cc:342] P efb59627dc9d4d75ab7f052e5bdbf070: Request end of tablet copy session 7105adb3d3484f50ba33155705b6b3e8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.137.125@KRBTEST.COM'} at 127.0.0.1:45120
I20260809 06:19:40.434737 10274 tablet_copy_service.cc:433] P efb59627dc9d4d75ab7f052e5bdbf070: ending tablet copy session 7105adb3d3484f50ba33155705b6b3e8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7105adb3d3484f50ba33155705b6b3e8
I20260809 06:19:40.446115 10760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:40.448907 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:40.452545 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:40.467258 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:40.467780 10760 server_base.cc:1034] running on GCE node
I20260809 06:19:40.468222 10760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:40.469064 10760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:40.470358 10760 hybrid_clock.cc:648] HybridClock initialized: now 1786256380470291 us; error 34 us; skew 500 ppm
I20260809 06:19:40.470679 10760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:40.473426 10760 init.cc:377] Logged in from keytab as oryx/127.3.137.125@KRBTEST.COM (short username oryx)
I20260809 06:19:40.474859 10760 webserver.cc:460] Webserver started at http://127.3.137.125:41917/ using document root <none> and password file <none>
I20260809 06:19:40.475286 10760 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:40.475436 10760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:40.478928 10760 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260809 06:19:40.481037 10893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:40.481480 10760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260809 06:19:40.481665 10760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "7105adb3d3484f50ba33155705b6b3e8"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "53b44ab2ee0b287eae8c19769dfc5791"
server_key_iv: "620e6bb93bee6d49b62d56133b101aec"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.481945 10760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:40.509941 10760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:40.514323 10760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:40.538283 10760 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.125:38263
I20260809 06:19:40.538316 10950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.125:38263 every 8 connection(s)
I20260809 06:19:40.539634 10760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
I20260809 06:19:40.555334 10951 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:19:40.559320 10951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Bootstrap starting.
I20260809 06:19:40.577637 10951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:19:40.578308 10951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: Bootstrap complete.
I20260809 06:19:40.579486 10951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } }
I20260809 06:19:40.579771 10951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7105adb3d3484f50ba33155705b6b3e8, State: Initialized, Role: LEARNER
I20260809 06:19:40.580035 10951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } }
I20260809 06:19:40.580979 10954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:40.581277 10951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:40.581359 10954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:19:40.597157 10966 catalog_manager.cc:1259] Loaded cluster ID: a0b918c1c940455382b3d40324d9b228
I20260809 06:19:40.597280 10966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: loading cluster ID for follower catalog manager: success
I20260809 06:19:40.601238 10966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: acquiring CA information for follower catalog manager: success
I20260809 06:19:40.604142 10966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:19:40.669096 10754 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
W20260809 06:19:40.766785 10872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:40.767347 10872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:40.767545 10872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:40.775044 10872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:40.775187 10872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:40.775280 10872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:40.775391 10872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:40.775478 10872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:40.785645 10872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.124.keytab
--principal=oryx/127.3.137.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:45459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.137.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:40.786998 10872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:40.788399 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:40.799996 10971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:40.799928 10972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:40.800429 10974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:40.800877 10872 server_base.cc:1034] running on GCE node
I20260809 06:19:40.801584 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:40.802917 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:40.804205 10872 hybrid_clock.cc:648] HybridClock initialized: now 1786256380804153 us; error 44 us; skew 500 ppm
I20260809 06:19:40.804605 10872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:40.807993 10872 init.cc:377] Logged in from keytab as oryx/127.3.137.124@KRBTEST.COM (short username oryx)
I20260809 06:19:40.809643 10872 webserver.cc:460] Webserver started at http://127.3.137.124:39301/ using document root <none> and password file <none>
I20260809 06:19:40.810379 10872 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:40.810534 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:40.810842 10872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:40.813988 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/instance:
uuid: "24c31bc7f7b2417eaf823fd275d3c984"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "36b27169c9308697a4eac4ebed59374d"
server_key_iv: "35f81f30fc49a80a8c532a449d870e3f"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.814855 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal/instance:
uuid: "24c31bc7f7b2417eaf823fd275d3c984"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "36b27169c9308697a4eac4ebed59374d"
server_key_iv: "35f81f30fc49a80a8c532a449d870e3f"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.820287 10872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:19:40.823637 10981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:40.824955 10872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:19:40.825139 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "24c31bc7f7b2417eaf823fd275d3c984"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "36b27169c9308697a4eac4ebed59374d"
server_key_iv: "35f81f30fc49a80a8c532a449d870e3f"
server_key_version: "encryptionkey@0"
I20260809 06:19:40.825349 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:40.862850 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:40.863010 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.856604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46840)
0809 06:19:40.857069 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.857092 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.857456 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.858059 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.858068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.858099 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:40.858287 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.858296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.859687 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.859692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.860918 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.860931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.862080 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.862086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.862260 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.862339 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.862454 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.862603 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260809 06:19:40.863010 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.856759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.3.137.125:38263)
0809 06:19:40.857459 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.857466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.857487 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:40.857588 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.857593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.857684 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.858005 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:40.858437 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.859532 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.861110 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.861890 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.862056 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.862270 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.862489 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.862646 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":350,"threads_started":1}
I20260809 06:19:40.864114 10922 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 06:19:40.864759 10954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb59627dc9d4d75ab7f052e5bdbf070. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:40.865054 10954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:19:40.868069 10872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:40.868402 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:40.891284 10872 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.124:45459
I20260809 06:19:40.891330 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.124:45459 every 8 connection(s)
I20260809 06:19:40.892719 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
I20260809 06:19:40.896574 11044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:40.900012 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 10872
I20260809 06:19:40.903829 11044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.124@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:40.923746 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.907652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.3.137.126:45079)
0809 06:19:40.907819 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.907823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.908471 (+ 648us) server_negotiation.cc:407] Connection header received
0809 06:19:40.908688 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.908696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.908819 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.908939 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.910306 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.911156 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.912997 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.913398 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.913493 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.917244 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.917249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.917252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.917282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.920497 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.921335 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.921340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.921342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.921476 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.921855 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.921858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.921860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.922213 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.922369 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.923127 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.923367 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:19:40.924280 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.902269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.3.137.124:45459)
0809 06:19:40.903079 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.903090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.903130 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:19:40.904851 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.904874 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.905251 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.905603 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.907509 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.909281 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.910870 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.911701 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.911830 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.916882 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.916896 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.916912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.916953 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.919779 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.920622 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.920630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.920638 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.920868 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.921225 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.921230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.921233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.921880 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.922098 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.922584 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.923061 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":358,"threads_started":1}
I20260809 06:19:40.924299 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.907569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45126)
0809 06:19:40.908286 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.908320 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.908549 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.909004 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.909013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.909801 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.910140 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.910156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.911324 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.911328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.912838 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.912856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.913637 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.913646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.913855 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.914467 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.914499 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.917062 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.920622 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.920631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.920641 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.921224 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.921584 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.921588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.921592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.921749 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.922375 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.922385 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.922580 (+ 195us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.922800 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.923186 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":65,"spinlock_wait_cycles":17152,"thread_start_us":387,"threads_started":1}
I20260809 06:19:40.925838 11044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.124@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:40.938072 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.926611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46846)
0809 06:19:40.926824 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.926843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.926947 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.927278 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.927281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.927600 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.927723 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.927730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.928754 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.928757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.929777 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.929788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.930515 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.930519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.930645 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.931259 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.931275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.933140 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.936099 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.936103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.936105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.936464 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.936808 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.936813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.936815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.936940 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.937410 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.937413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.937528 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.937635 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.937833 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 06:19:40.938158 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.926617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.3.137.125:38263)
0809 06:19:40.926798 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.926806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.926913 (+ 107us) server_negotiation.cc:407] Connection header received
0809 06:19:40.927102 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.927105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.927165 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.927252 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.927866 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.928637 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.929907 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.930392 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.930488 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.933309 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.933313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.933317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.933348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.935946 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.936583 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.936586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.936589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.936712 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.937032 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.937035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.937037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.937302 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.937413 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.937718 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.937897 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:40.939198 11044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:19:40 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256380, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.124@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
I20260809 06:19:40.951606 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.939923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.0.0.1:40406)
0809 06:19:40.940085 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.940111 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.940211 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.940534 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.940538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.940918 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.941068 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.941077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.941975 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.941979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.942807 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.942824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.943507 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.943512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:40.943685 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.944207 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:40.944226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:40.946202 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:40.949068 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.949071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.949074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.949414 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.949770 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:40.949773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:40.949775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:40.949926 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:40.950456 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:40.950461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:40.950590 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 06:19:40.950836 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 06:19:40.951144 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 06:19:40.951658 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.939968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.3.137.124:45459)
0809 06:19:40.940142 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.940146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.940162 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:19:40.940315 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.940319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.940407 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.940499 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.941194 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.941859 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.942934 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.943372 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.943471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:40.946370 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:40.946375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:40.946378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:40.946406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:40.948934 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.949555 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.949559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.949561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.949669 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:40.950038 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:40.950046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:40.950049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:40.950337 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:40.950493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:40.950922 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 06:19:40.951290 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":53}
I20260809 06:19:40.956408 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Bootstrap starting.
I20260809 06:19:40.959574 11044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:40.960625 11044 log.cc:824] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:40.963511 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: No bootstrap required, opened a new log
I20260809 06:19:40.970794 11044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } }
I20260809 06:19:40.971200 11044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:40.971354 11044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c31bc7f7b2417eaf823fd275d3c984, State: Initialized, Role: FOLLOWER
I20260809 06:19:40.971998 11044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } }
I20260809 06:19:40.972918 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } } }
I20260809 06:19:40.973237 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:40.973886 11044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 06:19:40.983820 11061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:19:40.984543 11061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:19:40.987737 10872 master_runner.cc:428] Detected that this master 24c31bc7f7b2417eaf823fd275d3c984 is joining an existing cluster
I20260809 06:19:40.987890 10872 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.137.124:45459
I20260809 06:19:41.013921 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.992075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.3.137.125:38263)
0809 06:19:40.992331 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.992338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:40.993592 (+ 1254us) server_negotiation.cc:407] Connection header received
0809 06:19:40.993947 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:40.993955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:40.994056 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:40.994142 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:40.996080 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.997545 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.000474 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.001389 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.001512 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.004895 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.004905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.004913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.004967 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.009512 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.010578 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.010587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.010593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.010796 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.011450 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.011459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.011464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.011935 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.012108 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.013025 (+ 917us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.013364 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 06:19:41.015388 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.991894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46858)
0809 06:19:40.993312 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:40.993336 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:40.993747 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:40.994415 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:40.994423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:40.995341 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:40.995861 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:40.995878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:40.997809 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:40.997817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.000251 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.000274 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.001927 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.001936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.002191 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.002998 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.003032 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.004611 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.009716 (+ 5105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.009725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.009735 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.010405 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.011015 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.011023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.011029 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.011287 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.012215 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.012223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.012545 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.014217 (+ 1672us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.014676 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":23256,"threads_started":1}
I20260809 06:19:41.017481 11050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:41.017828 11050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } }
I20260809 06:19:41.019722 11050 leader_election.cc:288] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079), 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125:38263)
I20260809 06:19:41.020748 10922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c31bc7f7b2417eaf823fd275d3c984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7105adb3d3484f50ba33155705b6b3e8" is_pre_election: true
I20260809 06:19:41.021445 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c31bc7f7b2417eaf823fd275d3c984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" is_pre_election: true
I20260809 06:19:41.022096 10985 leader_election.cc:302] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24c31bc7f7b2417eaf823fd275d3c984; no voters: 7105adb3d3484f50ba33155705b6b3e8, efb59627dc9d4d75ab7f052e5bdbf070
I20260809 06:19:41.022894 11050 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 06:19:41.030709 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10317
I20260809 06:19:41.033377 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.015821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.3.137.124:45459)
0809 06:19:41.016007 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.016013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.016196 (+ 183us) server_negotiation.cc:407] Connection header received
0809 06:19:41.016469 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.016473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.016543 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.016610 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.018056 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.019214 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.022132 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.023031 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.023155 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.026413 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.026418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.026421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.026448 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.030210 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.031108 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.031114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.031117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.031278 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.031721 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.031726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.031729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.032096 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.032243 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.032758 (+ 515us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.032991 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":67}
I20260809 06:19:41.034618 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.016007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.0.0.1:40420)
0809 06:19:41.016117 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.016137 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.016340 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.016805 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.016810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.017583 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.017876 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.017889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.019443 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.019447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.021976 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.021991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.024133 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.024139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.024308 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.024975 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.025002 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.026230 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.030400 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.030404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.030408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.030952 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.031425 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.031430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.031433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.031597 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.032326 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.032332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.032471 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.033933 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.034239 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":26}
I20260809 06:19:41.037595 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.991744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.3.137.126:45079)
0809 06:19:40.991965 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:40.991971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.015806 (+ 23835us) server_negotiation.cc:407] Connection header received
0809 06:19:41.016110 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.016118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.016211 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.016335 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.019850 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.020981 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.024068 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.025164 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.025297 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.028118 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.028125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.028133 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.028180 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.033102 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.034165 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.034174 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.034180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.034372 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.034994 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.035002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.035008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.035456 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.036229 (+ 773us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.036989 (+ 760us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.037233 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:19:41.038059 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:40.992507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45132)
0809 06:19:41.015540 (+ 23033us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.015563 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.015918 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.016600 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.016607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.017665 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.019644 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.019665 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.022090 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.022097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.023911 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.023925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.025571 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.025578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.025817 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.026479 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.026502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.027875 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.033304 (+ 5429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.033312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.033318 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.033983 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.034529 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.034537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.034544 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.034781 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.036316 (+ 1535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.036324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.036615 (+ 291us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.036797 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.037248 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22873,"mutex_wait_us":7,"thread_start_us":416,"threads_started":1}
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.65@KRBTEST.COM
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.126@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
I20260809 06:19:41.064369 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.032647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.3.137.65:40567)
0809 06:19:41.043518 (+ 10871us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.043527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.043557 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:19:41.043628 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.043638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.043917 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.044088 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.045782 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.046830 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.049100 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.050088 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.056386 (+ 6298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.056657 (+ 271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.056671 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.056691 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.056726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.060439 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.061292 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.061303 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.061306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.061476 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.061900 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.061906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.061910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.062365 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.062699 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.063645 (+ 946us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.063922 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10666,"thread_start_us":10494,"threads_started":1}
I20260809 06:19:41.065018 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.045066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.3.137.124:45459)
0809 06:19:41.048965 (+ 3899us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.048972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.048989 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:41.049074 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.049080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.049178 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.049276 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.050431 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.051520 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.053250 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.054043 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.054159 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.058324 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.058331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.058344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.058381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.062060 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.062892 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.062899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.062902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.063075 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.063541 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.063546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.063549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.063844 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.064019 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.064418 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.064642 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3767}
I20260809 06:19:41.065961 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.044379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.0.0.1:40424)
0809 06:19:41.044705 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.044723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.044831 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.049494 (+ 4663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.049499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.050029 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.050216 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.050230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.051682 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.051689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.053075 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.053091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.054379 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.054385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.054579 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.055334 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.055370 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.058122 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.062228 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.062235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.062238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.062763 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.063197 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.063201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.063205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.063370 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.063986 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.063991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.064174 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.064316 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.064598 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260809 06:19:41.066108 10454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:41.067672 10254 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.137.124:45459: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984"
member_type: NON_VOTER
last_known_addr {
host: "127.3.137.124"
port: 45459
}
attrs {
promote: true
}
}
dest_uuid: "efb59627dc9d4d75ab7f052e5bdbf070"
cas_config_opid_index: 5
I20260809 06:19:41.068673 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
I20260809 06:19:41.070616 11073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.071341 10922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 06:19:41.071575 11015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:41.072126 11072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 06:19:41.077268 11015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 06:19:41.079504 11072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:41.092428 11015 raft_consensus.cc:858] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
W20260809 06:19:41.092629 11015 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
I20260809 06:19:41.098673 10922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.104104 10954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.104447 10954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:19:41.105651 11073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.137.124:45459
I20260809 06:19:41.106294 10986 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.106643 10986 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.107425 10986 raft_consensus.cc:1050] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: attempting to promote NON_VOTER 7105adb3d3484f50ba33155705b6b3e8 to VOTER
I20260809 06:19:41.108243 10986 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
I20260809 06:19:41.109743 11073 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.110146 11073 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.111119 10922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 LEARNER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 06:19:41.111750 11072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 06:19:41.114223 11075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.118110 11075 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.118397 11075 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.118606 11072 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.118804 11072 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.118569 10922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.122702 10954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.122925 10954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:41.127421 10872 master.cc:537] Master@127.3.137.124:45459 shutting down...
W20260809 06:19:41.127599 10992 tablet.cc:2369] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:41.142573 11015 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:41.150450 11082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:41.150861 11082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 06:19:41.150743 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb59627dc9d4d75ab7f052e5bdbf070. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:41.151467 11014 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260809 06:19:41.151692 11014 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 06:19:41.151888 11079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: NON_VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: true } } }
I20260809 06:19:41.152086 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 06:19:41.152249 11079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 06:19:41.152774 11073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260809 06:19:41.157428 10237 consensus_peers.cc:487] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 -> Peer 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124:45459): Couldn't send request to peer 24c31bc7f7b2417eaf823fd275d3c984. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 06:19:41.157694 10872 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 06:19:41.158155 10872 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 06:19:41.158341 10872 tablet_replica.cc:330] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: stopping tablet replica
I20260809 06:19:41.165901 10872 master.cc:552] Master@127.3.137.124:45459 shutdown complete.
I20260809 06:19:41.166023 10872 master_runner.cc:305] Clearing existing system tablet
I20260809 06:19:41.166476 10872 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:41.166628 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:41.169831 10872 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 06:19:41.171535 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:41.171924 10872 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:41.172325 10872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 06:19:41.172472 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "24c31bc7f7b2417eaf823fd275d3c984"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "36b27169c9308697a4eac4ebed59374d"
server_key_iv: "35f81f30fc49a80a8c532a449d870e3f"
server_key_version: "encryptionkey@0"
I20260809 06:19:41.177312 10872 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:41.184132 10872 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 06:19:41.184391 10872 log.cc:1196] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 06:19:41.184959 10872 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Deleting consensus metadata
I20260809 06:19:41.185247 10872 master_runner.cc:315] Copying system tablet from 127.3.137.126:45079
I20260809 06:19:41.192734 10872 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: tablet copy: Beginning tablet copy session from remote peer at address 127.3.137.126:45079
I20260809 06:19:41.214262 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10317
I20260809 06:19:41.215132 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.193698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45144)
0809 06:19:41.199145 (+ 5447us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.199167 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.199332 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.199754 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.199759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.200279 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.200630 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.200642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.202212 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.202218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.203733 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.203759 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.204829 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.204835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.205015 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.205801 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.205823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.207007 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.211005 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.211010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.211015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.211547 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.212996 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.213001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.213004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.213182 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.213724 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.213731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.213949 (+ 218us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.214109 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.214484 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5198,"thread_start_us":5051,"threads_started":1}
I20260809 06:19:41.216225 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.193773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.3.137.126:45079)
0809 06:19:41.194061 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.194068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.199295 (+ 5227us) server_negotiation.cc:407] Connection header received
0809 06:19:41.199486 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.199492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.199579 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.199695 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.200812 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.202039 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.203935 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.204617 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.204756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.207201 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.207208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.207215 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.207255 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.210842 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.212320 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.212345 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.212391 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.212839 (+ 448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.213336 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.213340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.213342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.213602 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.213768 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.215487 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.215755 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260809 06:19:41.216845 10274 tablet_copy_service.cc:140] P efb59627dc9d4d75ab7f052e5bdbf070: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 24c31bc7f7b2417eaf823fd275d3c984 ({username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:45144)
I20260809 06:19:41.217026 10274 tablet_copy_service.cc:161] P efb59627dc9d4d75ab7f052e5bdbf070: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 24c31bc7f7b2417eaf823fd275d3c984 at {username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:45144: session id = 24c31bc7f7b2417eaf823fd275d3c984-00000000000000000000000000000000
I20260809 06:19:41.220674 10274 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 06:19:41.222607 10872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 06:19:41.230734 10872 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: tablet copy: Starting download of 0 data blocks...
I20260809 06:19:41.231145 10872 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: tablet copy: Starting download of 1 WAL segments...
I20260809 06:19:41.233731 10872 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 06:19:41.238610 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:40567
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:45079
--webserver_port=40881
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:19:41.243278 10274 tablet_copy_service.cc:342] P efb59627dc9d4d75ab7f052e5bdbf070: Request end of tablet copy session 24c31bc7f7b2417eaf823fd275d3c984-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:45144
I20260809 06:19:41.243428 10274 tablet_copy_service.cc:433] P efb59627dc9d4d75ab7f052e5bdbf070: ending tablet copy session 24c31bc7f7b2417eaf823fd275d3c984-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 24c31bc7f7b2417eaf823fd275d3c984
I20260809 06:19:41.255162 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:41.257750 11094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:41.260246 11095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:41.267977 10872 server_base.cc:1034] running on GCE node
W20260809 06:19:41.268219 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:41.268821 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:41.269591 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:41.270853 10872 hybrid_clock.cc:648] HybridClock initialized: now 1786256381270810 us; error 27 us; skew 500 ppm
I20260809 06:19:41.271145 10872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:41.273351 10872 init.cc:377] Logged in from keytab as oryx/127.3.137.124@KRBTEST.COM (short username oryx)
I20260809 06:19:41.276577 10872 webserver.cc:460] Webserver started at http://127.3.137.124:37449/ using document root <none> and password file <none>
I20260809 06:19:41.276932 10872 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:41.277042 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:41.280247 10872 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260809 06:19:41.282204 11105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:41.282714 10872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260809 06:19:41.282881 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "24c31bc7f7b2417eaf823fd275d3c984"
format_stamp: "Formatted at 2026-08-09 06:19:40 on dist-test-slave-2w3w"
server_key: "36b27169c9308697a4eac4ebed59374d"
server_key_iv: "35f81f30fc49a80a8c532a449d870e3f"
server_key_version: "encryptionkey@0"
I20260809 06:19:41.283134 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:41.321959 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:41.326449 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:41.354734 10872 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.124:45459
I20260809 06:19:41.354820 11161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.124:45459 every 8 connection(s)
I20260809 06:19:41.356017 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
I20260809 06:19:41.361507 11162 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:19:41.365692 11162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Bootstrap starting.
I20260809 06:19:41.380894 11162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:19:41.381449 11162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Bootstrap complete.
I20260809 06:19:41.382570 11162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
I20260809 06:19:41.382891 11162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 24c31bc7f7b2417eaf823fd275d3c984, State: Initialized, Role: LEARNER
I20260809 06:19:41.383268 11162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } }
I20260809 06:19:41.384336 11166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.384788 11162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:41.384995 11166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:19:41.401156 11180 catalog_manager.cc:1259] Loaded cluster ID: a0b918c1c940455382b3d40324d9b228
I20260809 06:19:41.401273 11180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: loading cluster ID for follower catalog manager: success
I20260809 06:19:41.404973 11180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: acquiring CA information for follower catalog manager: success
I20260809 06:19:41.407539 11180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 06:19:41.606927 11093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:41.607406 11093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:41.607573 11093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:41.615233 11093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:41.615378 11093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:41.615546 11093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:41.626185 11093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:40567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=40881
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:41.627648 11093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:41.628952 11093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:41.643131 11186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:41.643177 11190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:41.643873 11188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:41.644050 11093 server_base.cc:1034] running on GCE node
I20260809 06:19:41.644675 11093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:41.645644 11093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:41.646875 11093 hybrid_clock.cc:648] HybridClock initialized: now 1786256381646827 us; error 54 us; skew 500 ppm
I20260809 06:19:41.647213 11093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:41.650513 11093 init.cc:377] Logged in from keytab as oryx/127.3.137.65@KRBTEST.COM (short username oryx)
I20260809 06:19:41.652027 11093 webserver.cc:460] Webserver started at http://127.3.137.65:40881/ using document root <none> and password file <none>
I20260809 06:19:41.652771 11093 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:41.652926 11093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:41.658219 11093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 06:19:41.660808 11197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:41.662086 11093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:41.662221 11093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "589fed3fea814338b69691e8e33a08d7"
format_stamp: "Formatted at 2026-08-09 06:19:38 on dist-test-slave-2w3w"
server_key: "c544717a07849735b6f3924cab7b8d63"
server_key_iv: "6dfb52009191db6e57898452f26500d8"
server_key_version: "encryptionkey@0"
I20260809 06:19:41.663236 11093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:41.681478 11093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:41.686406 11093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:41.686650 11093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:41.687704 11093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:41.689419 11093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:41.689514 11093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:41.689600 11093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:41.689657 11093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:41.727097 11093 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40567
I20260809 06:19:41.727115 11317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40567 every 8 connection(s)
I20260809 06:19:41.728662 11093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:41.737063 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 11093
I20260809 06:19:41.737447 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10465
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
Aug 09 06:19:41 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.66@KRBTEST.COM
I20260809 06:19:41.771533 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.753708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.0.0.1:40430)
0809 06:19:41.762831 (+ 9123us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.762849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.763025 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.763699 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.763703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.763731 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:41.764058 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.764072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.766016 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.766024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.768409 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.768424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.770645 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.770651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.770842 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.770894 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.771029 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.771185 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9004,"thread_start_us":8893,"threads_started":1}
I20260809 06:19:41.772387 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.754209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.3.137.124:45459)
0809 06:19:41.757070 (+ 2861us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.757075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.762953 (+ 5878us) server_negotiation.cc:407] Connection header received
0809 06:19:41.763161 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.763167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.763349 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.763471 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:41.764262 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.765830 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.768605 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.770122 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.770266 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.770528 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.771831 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.771999 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2728,"thread_start_us":2607,"threads_started":1}
I20260809 06:19:41.774226 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.736180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:44699 (local address 127.3.137.124:45459)
0809 06:19:41.751935 (+ 15755us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.751941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.751959 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:19:41.752012 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.752018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.752122 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.752238 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.754128 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.756009 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.758574 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.759492 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.759639 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.764634 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.764638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.764642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.764672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.768956 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.769949 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.769956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.769962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.770127 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.770622 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.770630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.770637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.771039 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.771191 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.773581 (+ 2390us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.773814 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15593,"thread_start_us":15466,"threads_started":1}
I20260809 06:19:41.774617 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.731624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:53841 (local address 127.3.137.125:38263)
0809 06:19:41.733455 (+ 1831us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.733465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.737065 (+ 3600us) server_negotiation.cc:407] Connection header received
0809 06:19:41.741049 (+ 3984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.741059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.741193 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.741319 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.744472 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.746015 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.751792 (+ 5777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.752485 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.752622 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.767348 (+ 14726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.767355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.767360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.767404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.771356 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.772257 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.772263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.772266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.772448 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.772950 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.772955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.772958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.773294 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.773412 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.773990 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.774212 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1655,"thread_start_us":1533,"threads_started":1}
I20260809 06:19:41.775203 11134 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260809 06:19:41.776065 11166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb59627dc9d4d75ab7f052e5bdbf070. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: NON_VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: true } } }
I20260809 06:19:41.776379 11166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:19:41.776368 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.736053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.3.137.65:44699)
0809 06:19:41.738300 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.738325 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.740736 (+ 2411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.752291 (+ 11555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.752296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.753626 (+ 1330us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.753937 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.753951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.756197 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.756205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.758318 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.758340 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.759977 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.759986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.760265 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.760953 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.760985 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.764312 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.769123 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.769132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.769151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.769796 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.770288 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.770296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.770300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.770497 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.771290 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.771301 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.771775 (+ 474us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.772185 (+ 410us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.772777 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"thread_start_us":311,"threads_started":1}
I20260809 06:19:41.776755 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.732392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.3.137.65:53841)
0809 06:19:41.736713 (+ 4321us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.736732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.740838 (+ 4106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.741903 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.741920 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.743140 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.744195 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.744233 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.747416 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.747420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.751574 (+ 4154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.751605 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.752912 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.752919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.753182 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.762268 (+ 9086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.762297 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.767082 (+ 4785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.771525 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.771531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.771534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.772115 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.772618 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.772623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.772627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.772828 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.773508 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.773512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.773673 (+ 161us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.774836 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.775167 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4144,"mutex_wait_us":448,"thread_start_us":2256,"threads_started":1}
I20260809 06:19:41.778152 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.732392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:45879)
0809 06:19:41.735436 (+ 3044us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:41.735474 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:41.740277 (+ 4803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:41.746258 (+ 5981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:41.746266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:41.746866 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:41.747228 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.747239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.749547 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.749556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.751577 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:41.751604 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.758811 (+ 7207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.758818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:41.759010 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.759722 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:41.759757 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:41.764961 (+ 5204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:41.770672 (+ 5711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.770678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.770683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.771207 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.774811 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:41.774815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:41.774818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:41.775003 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:41.776290 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:41.776295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:41.776538 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 06:19:41.777154 (+ 616us) client_negotiation.cc:240] Negotiation successful
0809 06:19:41.777525 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2786,"mutex_wait_us":56,"thread_start_us":6774,"threads_started":1}
I20260809 06:19:41.778641 11321 heartbeater.cc:347] Connected to a master server at 127.3.137.125:38263
I20260809 06:19:41.778775 11319 heartbeater.cc:347] Connected to a master server at 127.3.137.124:45459
I20260809 06:19:41.779115 11321 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:41.779083 11319 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:41.780458 11319 heartbeater.cc:511] Master 127.3.137.124:45459 requested a full tablet report, sending...
I20260809 06:19:41.780728 11321 heartbeater.cc:511] Master 127.3.137.125:38263 requested a full tablet report, sending...
I20260809 06:19:41.780820 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.731941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:45879 (local address 127.3.137.126:45079)
0809 06:19:41.733886 (+ 1945us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.733894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.735632 (+ 1738us) server_negotiation.cc:407] Connection header received
0809 06:19:41.742203 (+ 6571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.742211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.742329 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.742651 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.747378 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.748841 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.751825 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.752495 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.752626 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.765190 (+ 12564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.765197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.765202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.765253 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.770191 (+ 4938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.774452 (+ 4261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.774461 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.774467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.774638 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.775149 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.775154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.775157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.775614 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.775752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.780063 (+ 4311us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.780319 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":1668,"threads_started":1}
I20260809 06:19:41.781836 11073 raft_consensus.cc:1050] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: attempting to promote NON_VOTER 24c31bc7f7b2417eaf823fd275d3c984 to VOTER
I20260809 06:19:41.782676 11073 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } }
I20260809 06:19:41.783185 10911 ts_manager.cc:194] Registered new tserver with Master: 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:41.783921 11320 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:41.784057 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:41.738527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.3.137.66:37993)
0809 06:19:41.751038 (+ 12511us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:41.751048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:41.751452 (+ 404us) server_negotiation.cc:407] Connection header received
0809 06:19:41.752715 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:41.752725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:41.753018 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:41.753152 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:41.756004 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.757414 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.758672 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:41.759639 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:41.759787 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:41.772353 (+ 12566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:41.772368 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:41.772384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:41.772435 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:41.779352 (+ 6917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.780410 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.780419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.780423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.780617 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:41.781138 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:41.781144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:41.781147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:41.781598 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:41.782145 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:41.782782 (+ 637us) server_negotiation.cc:299] Negotiation successful
0809 06:19:41.783393 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12308,"thread_start_us":12540,"threads_started":1}
I20260809 06:19:41.784070 11320 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:41.784653 11320 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
I20260809 06:19:41.785794 10598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:41.785979 10254 ts_manager.cc:194] Re-registered known tserver with Master: 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:41.786229 11123 ts_manager.cc:194] Registered new tserver with Master: 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:41.786484 10922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 06:19:41.786933 11134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 LEARNER]: Refusing update from remote peer efb59627dc9d4d75ab7f052e5bdbf070: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 06:19:41.787117 11073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 06:19:41.787531 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:45879
I20260809 06:19:41.788780 11335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 06:19:41.790226 11073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.792189 10922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.795670 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.795989 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.795257 11134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.797794 11334 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.798350 11334 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.798624 11073 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.798919 11073 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:41.800566 11339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.801434 11339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:41.802245 11166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:41.802552 11166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:41.891338 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10465
I20260809 06:19:41.911443 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:37993
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:45079
--webserver_port=44613
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:42.253643 11341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:42.253996 11341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:42.254155 11341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:42.261341 11341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:42.261483 11341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:42.261658 11341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:42.271597 11341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:37993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=44613
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:42.272923 11341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:42.274395 11341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:42.294220 11346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:42.294260 11347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:42.294294 11349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:42.295359 11341 server_base.cc:1034] running on GCE node
I20260809 06:19:42.296003 11341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:42.297034 11341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:42.298273 11341 hybrid_clock.cc:648] HybridClock initialized: now 1786256382298197 us; error 61 us; skew 500 ppm
I20260809 06:19:42.298604 11341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:42.302213 11341 init.cc:377] Logged in from keytab as oryx/127.3.137.66@KRBTEST.COM (short username oryx)
I20260809 06:19:42.304651 11341 webserver.cc:460] Webserver started at http://127.3.137.66:44613/ using document root <none> and password file <none>
I20260809 06:19:42.305363 11341 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:42.305523 11341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:42.310817 11341 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:19:42.313992 11356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:42.315562 11341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:42.315739 11341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "efba900d3fac4e4dbf5f89860749f96c"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "6a07d4bace350c78567bb345711c038c"
server_key_iv: "b6614c653d36201a57f0a8c049a9e145"
server_key_version: "encryptionkey@0"
I20260809 06:19:42.316709 11341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:42.332046 11341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:42.335997 11341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:42.336275 11341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:42.337239 11341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:42.338557 11341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:42.338682 11341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:42.338899 11341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:42.339018 11341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:42.376513 11341 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:37993
I20260809 06:19:42.376596 11473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:37993 every 8 connection(s)
I20260809 06:19:42.378078 11341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:42.385115 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 11341
I20260809 06:19:42.385476 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10609
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:42.422181 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.380987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:52809 (local address 127.3.137.124:45459)
0809 06:19:42.387409 (+ 6422us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:42.387413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:42.390064 (+ 2651us) server_negotiation.cc:407] Connection header received
0809 06:19:42.394203 (+ 4139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:42.394214 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:42.394348 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:42.394474 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:42.397322 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.398839 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.407159 (+ 8320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.408264 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.408359 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.413829 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:42.413836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:42.413841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:42.413890 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:42.417576 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.418739 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.418745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.418748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.418930 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.419740 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.419746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.419752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.420173 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:42.420331 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:42.421547 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0809 06:19:42.421786 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6256,"thread_start_us":6138,"threads_started":1}
I20260809 06:19:42.423966 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.382459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:58635)
0809 06:19:42.393860 (+ 11401us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:42.393885 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:42.394019 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:42.401891 (+ 7872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:42.401899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:42.402749 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:42.403156 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.403173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.406534 (+ 3361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.406538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.408034 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.408049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.409080 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.409086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.409283 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.410057 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:42.410076 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:42.413942 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:42.417811 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.417815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.417818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.418597 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.419827 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.419832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.419837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.420010 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.420678 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:42.420684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:42.421031 (+ 347us) client_negotiation.cc:769] Sending connection context
0809 06:19:42.421399 (+ 368us) client_negotiation.cc:240] Negotiation successful
0809 06:19:42.421953 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11234,"spinlock_wait_cycles":24192,"thread_start_us":11037,"threads_started":1}
I20260809 06:19:42.424008 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.381682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.3.137.66:52809)
0809 06:19:42.389905 (+ 8223us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:42.389949 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:42.392910 (+ 2961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:42.395057 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:42.395074 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:42.396074 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:42.397082 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.397114 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.399131 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.399140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.406350 (+ 7210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.406365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.408700 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.408707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.409016 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.409904 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:42.409954 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:42.413521 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:42.417748 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.417770 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.417787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.418597 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.419308 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.419315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.419320 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.419613 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.420420 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:42.420428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:42.421034 (+ 606us) client_negotiation.cc:769] Sending connection context
0809 06:19:42.422374 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0809 06:19:42.422675 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7938,"spinlock_wait_cycles":1408,"thread_start_us":7730,"threads_started":1}
I20260809 06:19:42.426016 11474 heartbeater.cc:347] Connected to a master server at 127.3.137.124:45459
I20260809 06:19:42.426358 11474 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:42.425921 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.381016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58635 (local address 127.3.137.126:45079)
0809 06:19:42.401438 (+ 20422us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:42.401447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:42.401475 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:19:42.401553 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:42.401562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:42.401672 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:42.403364 (+ 1692us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:42.403582 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.405015 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.408214 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.408865 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.409544 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.414173 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:42.414181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:42.414186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:42.414223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:42.417659 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.418745 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.418750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.418753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.419169 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.420150 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.420155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.420158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.420533 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:42.425128 (+ 4595us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:42.425274 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 06:19:42.425517 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20236,"thread_start_us":20079,"threads_started":1}
I20260809 06:19:42.427388 11474 heartbeater.cc:511] Master 127.3.137.124:45459 requested a full tablet report, sending...
I20260809 06:19:42.428380 11475 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:42.428555 11475 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:42.429052 11475 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
I20260809 06:19:42.429315 11123 ts_manager.cc:194] Registered new tserver with Master: efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:42.430168 10254 ts_manager.cc:194] Re-registered known tserver with Master: efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.67@KRBTEST.COM
I20260809 06:19:42.431505 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:58635
I20260809 06:19:42.438009 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.390933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:57237 (local address 127.3.137.125:38263)
0809 06:19:42.406481 (+ 15548us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:42.406490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:42.419807 (+ 13317us) server_negotiation.cc:407] Connection header received
0809 06:19:42.420067 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:42.420075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:42.420173 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:42.420257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:42.421707 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.422825 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.424822 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.425412 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.425505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.431961 (+ 6456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:42.431967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:42.431973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:42.432018 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:42.435393 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.436054 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.436059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.436063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.436205 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.436613 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.436618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.436621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.436900 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:42.437060 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:42.437416 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 06:19:42.437627 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15402,"thread_start_us":15528,"threads_started":1}
I20260809 06:19:42.438052 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.390780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.3.137.66:57237)
0809 06:19:42.419696 (+ 28916us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:42.419716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:42.419905 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:42.420485 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:42.420490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:42.421168 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:42.421541 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.421553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.423041 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.423047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.424639 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:42.424657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.425831 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.425839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:42.426057 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.426643 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:42.426665 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:42.431698 (+ 5033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:42.435550 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.435555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.435559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.435921 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.436338 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:42.436343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:42.436347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:42.436496 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:42.437020 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:42.437025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:42.437166 (+ 141us) client_negotiation.cc:769] Sending connection context
0809 06:19:42.437308 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 06:19:42.437630 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28774,"thread_start_us":28600,"threads_started":1}
I20260809 06:19:42.438652 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:42.387087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.3.137.67:45815)
0809 06:19:42.414560 (+ 27473us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:42.414573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:42.415789 (+ 1216us) server_negotiation.cc:407] Connection header received
0809 06:19:42.415873 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:42.415885 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:42.416172 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:42.416551 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:42.419547 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.421093 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.422990 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:42.423847 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:42.423983 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:42.432121 (+ 8138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:42.432133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:42.432146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:42.432184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:42.435381 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.436097 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.436104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.436106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.436248 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:42.436664 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:42.436668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:42.436670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:42.436992 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:42.437309 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:42.437629 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 06:19:42.438299 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27211,"thread_start_us":27468,"threads_started":1}
I20260809 06:19:42.439287 11476 heartbeater.cc:347] Connected to a master server at 127.3.137.125:38263
I20260809 06:19:42.439430 11476 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:42.439877 11476 heartbeater.cc:511] Master 127.3.137.125:38263 requested a full tablet report, sending...
I20260809 06:19:42.440300 10749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:42.441011 10911 ts_manager.cc:194] Registered new tserver with Master: efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
I20260809 06:19:42.523891 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10609
I20260809 06:19:42.541640 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:45815
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:45079
--webserver_port=39773
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:19:42.791095 11320 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
W20260809 06:19:42.880501 11489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:42.880828 11489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:42.881006 11489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:42.889426 11489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:42.889580 11489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:42.889758 11489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:42.899607 11489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:45815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=39773
--tserver_master_addrs=127.3.137.126:45079,127.3.137.125:38263,127.3.137.124:45459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:42.900949 11489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:42.902374 11489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:42.915916 11499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:42.916100 11495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:42.916043 11497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:42.920441 11489 server_base.cc:1034] running on GCE node
I20260809 06:19:42.921099 11489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:42.922096 11489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:42.923341 11489 hybrid_clock.cc:648] HybridClock initialized: now 1786256382923282 us; error 52 us; skew 500 ppm
I20260809 06:19:42.923651 11489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:42 dist-test-slave-2w3w krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:42.926788 11489 init.cc:377] Logged in from keytab as oryx/127.3.137.67@KRBTEST.COM (short username oryx)
I20260809 06:19:42.928318 11489 webserver.cc:460] Webserver started at http://127.3.137.67:39773/ using document root <none> and password file <none>
I20260809 06:19:42.929030 11489 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:42.929189 11489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:42.934413 11489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:19:42.937112 11509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:42.938546 11489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:42.938752 11489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
format_stamp: "Formatted at 2026-08-09 06:19:39 on dist-test-slave-2w3w"
server_key: "aab95a02b3d51bcf028dc8d4d4fe9328"
server_key_iv: "cc8df043913ddf08e2471160ac3b1109"
server_key_version: "encryptionkey@0"
I20260809 06:19:42.939854 11489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:42.980295 11489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:42.984901 11489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:42.985201 11489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:42.986228 11489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:42.987847 11489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:42.987972 11489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:42.988097 11489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:42.988161 11489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:43.023176 11489 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:45815
I20260809 06:19:43.023254 11626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:45815 every 8 connection(s)
I20260809 06:19:43.024652 11489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:43.026372 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 11489
Aug 09 06:19:43 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:43.069420 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.028200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:51261)
0809 06:19:43.029228 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.029285 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.032684 (+ 3399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.036215 (+ 3531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.036233 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.037236 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:43.038241 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.038273 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.040727 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.040736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.043263 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.043286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.053013 (+ 9727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.053023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.053374 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.054386 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:43.054424 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:43.059634 (+ 5210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:43.064137 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.064146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.064164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.064897 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.065436 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.065444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.065450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.065727 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.066568 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:43.066578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:43.067069 (+ 491us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.067369 (+ 300us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.067907 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":879,"threads_started":1}
I20260809 06:19:43.070556 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.028883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:51261 (local address 127.3.137.126:45079)
0809 06:19:43.035385 (+ 6502us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.035392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.035415 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:19:43.035487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.035493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.035584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.035727 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.038493 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.040502 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.043480 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.044296 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.049668 (+ 5372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.059919 (+ 10251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.059929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.059936 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.059992 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.063944 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.065082 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.065091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.065097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.065293 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.065876 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.065882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.065887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.066309 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.066472 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.069936 (+ 3464us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.070192 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6355,"thread_start_us":7073,"threads_started":1}
I20260809 06:19:43.073588 11628 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45079
I20260809 06:19:43.074441 11628 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:43.075604 11628 heartbeater.cc:511] Master 127.3.137.126:45079 requested a full tablet report, sending...
I20260809 06:19:43.077656 10254 ts_manager.cc:194] Re-registered known tserver with Master: 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815)
I20260809 06:19:43.078555 10254 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:51261
Aug 09 06:19:43 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.124@KRBTEST.COM
Aug 09 06:19:43 dist-test-slave-2w3w krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.125@KRBTEST.COM
I20260809 06:19:43.091506 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.028499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:43703 (local address 127.3.137.124:45459)
0809 06:19:43.036248 (+ 7749us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.036256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.052506 (+ 16250us) server_negotiation.cc:407] Connection header received
0809 06:19:43.061020 (+ 8514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.061027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.061140 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.061231 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.066621 (+ 5390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.067746 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.072801 (+ 5055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.073505 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.073612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.081697 (+ 8085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.081703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.081707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.081766 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.085900 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.089237 (+ 3337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.089243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.089247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.089394 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.089850 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.089856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.089860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.090225 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.090368 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.090896 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.091147 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7621,"thread_start_us":13788,"threads_started":1}
I20260809 06:19:43.091826 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.036758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.3.137.126:45079)
0809 06:19:43.053023 (+ 16265us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.053028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.053045 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:43.053095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.053100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.053172 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.053278 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.062899 (+ 9621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.064156 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.071755 (+ 7599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.072438 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.072573 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.084148 (+ 11575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.084154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.084159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.084199 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.087656 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.088718 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.088725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.088730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.088879 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.089509 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.089515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.089519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.089884 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.090119 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.091218 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.091444 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16148,"thread_start_us":18320,"threads_started":1}
I20260809 06:19:43.092891 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.028973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.3.137.67:43703)
0809 06:19:43.052377 (+ 23404us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.052398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.052560 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.061597 (+ 9037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.061602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.062237 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:43.062569 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.062586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.068093 (+ 5507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.068098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.069657 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.069672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.077185 (+ 7513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.077194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.077406 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.078470 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:43.078492 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:43.081484 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:43.086060 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.086067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.086072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.089096 (+ 3024us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.089546 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.089552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.089558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.089703 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.090454 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:43.090459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:43.090615 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.092195 (+ 1580us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.092481 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23251,"thread_start_us":25206,"threads_started":1}
I20260809 06:19:43.093865 11627 heartbeater.cc:347] Connected to a master server at 127.3.137.124:45459
I20260809 06:19:43.094005 11627 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:43.094457 11627 heartbeater.cc:511] Master 127.3.137.124:45459 requested a full tablet report, sending...
I20260809 06:19:43.095847 11123 ts_manager.cc:194] Registered new tserver with Master: 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815)
I20260809 06:19:43.096357 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.054837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47875 (local address 127.3.137.125:38263)
0809 06:19:43.067538 (+ 12701us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.067547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.072603 (+ 5056us) server_negotiation.cc:407] Connection header received
0809 06:19:43.072663 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.072668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.072743 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.072843 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.075198 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.076408 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.078823 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.079640 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.079770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.086924 (+ 7154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.086930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.086935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.087003 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.091009 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.091771 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.091777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.091781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.091940 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.092386 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.092392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.092397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.093511 (+ 1114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.093659 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.095753 (+ 2094us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.095977 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":740,"server-negotiator.queue_time_us":12515,"thread_start_us":13073,"threads_started":1}
I20260809 06:19:43.096776 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.060436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.3.137.67:47875)
0809 06:19:43.071398 (+ 10962us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.071422 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.071548 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.073102 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.073107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.073802 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:43.075011 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.075028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.076713 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.076718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.078662 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.078679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.082133 (+ 3454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.082141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.082322 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.083173 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:43.083199 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:43.086693 (+ 3494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:43.091160 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.091165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.091169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.091623 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.092085 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:43.092090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:43.092094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:43.092259 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:43.095289 (+ 3030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:43.095297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:43.095473 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.095635 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.096350 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10837,"thread_start_us":20184,"threads_started":1}
I20260809 06:19:43.097645 11629 heartbeater.cc:347] Connected to a master server at 127.3.137.125:38263
I20260809 06:19:43.097780 11629 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:43.098136 11629 heartbeater.cc:511] Master 127.3.137.125:38263 requested a full tablet report, sending...
I20260809 06:19:43.099272 10911 ts_manager.cc:194] Registered new tserver with Master: 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815)
I20260809 06:19:43.096597 10254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 06:19:43.106056 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.037188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.3.137.125:38263)
0809 06:19:43.049266 (+ 12078us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.049274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.070435 (+ 21161us) server_negotiation.cc:407] Connection header received
0809 06:19:43.071896 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.071905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.072024 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.072125 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.087009 (+ 14884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.088444 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.092593 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.093108 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.093201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.099510 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.099518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.099528 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.099572 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.103054 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.103824 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.103829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.103833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.104018 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.104434 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.104438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.104443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.104780 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.105022 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.105434 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.105654 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":11944,"thread_start_us":11811,"threads_started":1}
I20260809 06:19:43.106245 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.050715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.3.137.124:45459)
0809 06:19:43.079784 (+ 29069us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.079790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.079807 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:43.079857 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.079862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.079939 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.080080 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.082602 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.083883 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.088677 (+ 4794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.089274 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.089404 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.098168 (+ 8764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.098177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.098183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.098284 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.102449 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.103316 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.103322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.103326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.103455 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.103894 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.103900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.103903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.104328 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.104577 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.105572 (+ 995us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.105843 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28915,"thread_start_us":29518,"threads_started":1}
W20260809 06:19:43.106513 10254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:43.147910 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.126763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45815 (local address 127.0.0.1:37862)
0809 06:19:43.128314 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.128337 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.128512 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.136430 (+ 7918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.136439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.136477 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:43.136999 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.137015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.142434 (+ 5419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.142443 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.144787 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.144800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.146917 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.146924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.147127 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.147171 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.147300 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.147452 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"thread_start_us":13248,"threads_started":1}
I20260809 06:19:43.147905 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.127074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.3.137.67:45815)
0809 06:19:43.135557 (+ 8483us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.135571 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.135626 (+ 55us) server_negotiation.cc:407] Connection header received
0809 06:19:43.135759 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.135772 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.136139 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.137256 (+ 1117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:43.139061 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.142239 (+ 3178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.145132 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.146693 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.146820 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.147106 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.147359 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.147553 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8270,"thread_start_us":8104,"threads_started":1}
I20260809 06:19:43.151849 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.127838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.3.137.66:37993)
0809 06:19:43.144343 (+ 16505us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.144351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.144375 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:19:43.144469 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.144474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.144657 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.144870 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:43.145445 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.146904 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.149409 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.150778 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.150916 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.151151 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.151410 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.151575 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16307,"thread_start_us":14707,"threads_started":1}
I20260809 06:19:43.151865 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.127508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:37993 (local address 127.0.0.1:59840)
0809 06:19:43.140364 (+ 12856us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.140394 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.140596 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.144858 (+ 4262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.144862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.144887 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:43.145265 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.145277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.147114 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.147120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.149240 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.149255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.151065 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.151070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.151200 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.151225 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.151347 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.151501 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12682,"thread_start_us":415,"threads_started":1}
I20260809 06:19:43.152259 11557 tablet_service.cc:1467] Processing CreateTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b (DEFAULT_TABLE table=test-table [id=847924f0bde443e18450eedd382c9add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:43.153546 11557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed94a14f41f04d7b8b51b698a8d9930b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:43.157090 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.126023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40567 (local address 127.0.0.1:50340)
0809 06:19:43.126740 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.126759 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.127013 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.144015 (+ 17002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.144020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.144046 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:43.144433 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.144445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.145918 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.145924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.147687 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.147700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.156224 (+ 8524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.156232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.156416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.156462 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.156533 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.156675 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
I20260809 06:19:43.157773 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.126311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.3.137.65:40567)
0809 06:19:43.143358 (+ 17047us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.143366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.143401 (+ 35us) server_negotiation.cc:407] Connection header received
0809 06:19:43.143511 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.143521 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.143794 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.144005 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:43.144645 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.145774 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.147888 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.155726 (+ 7838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.157000 (+ 1274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.157252 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.157322 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.157499 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16878,"thread_start_us":16804,"threads_started":1}
I20260809 06:19:43.158660 11405 tablet_service.cc:1467] Processing CreateTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b (DEFAULT_TABLE table=test-table [id=847924f0bde443e18450eedd382c9add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:43.160146 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed94a14f41f04d7b8b51b698a8d9930b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:43.160734 11246 tablet_service.cc:1467] Processing CreateTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b (DEFAULT_TABLE table=test-table [id=847924f0bde443e18450eedd382c9add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:43.161794 11246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed94a14f41f04d7b8b51b698a8d9930b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:43.169405 11665 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Bootstrap starting.
I20260809 06:19:43.173336 11665 tablet_bootstrap.cc:654] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:43.174665 11665 log.cc:824] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:43.174767 11666 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Bootstrap starting.
I20260809 06:19:43.176242 11667 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Bootstrap starting.
I20260809 06:19:43.178654 11666 tablet_bootstrap.cc:654] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:43.179877 11667 tablet_bootstrap.cc:654] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:43.180023 11666 log.cc:824] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:43.181160 11667 log.cc:824] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:43.181910 11665 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: No bootstrap required, opened a new log
I20260809 06:19:43.182253 11665 ts_tablet_manager.cc:1397] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260809 06:19:43.183578 11666 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: No bootstrap required, opened a new log
I20260809 06:19:43.183852 11666 ts_tablet_manager.cc:1397] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260809 06:19:43.185034 11667 tablet_bootstrap.cc:492] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: No bootstrap required, opened a new log
I20260809 06:19:43.185312 11667 ts_tablet_manager.cc:1397] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 06:19:43.191051 11665 raft_consensus.cc:348] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.191433 11665 raft_consensus.cc:374] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:43.191555 11665 raft_consensus.cc:729] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cb0dbd19c3490baa9657a7ca1b46ff, State: Initialized, Role: FOLLOWER
I20260809 06:19:43.192224 11665 consensus_queue.cc:260] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.194310 11628 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
I20260809 06:19:43.194768 11666 raft_consensus.cc:348] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.195226 11666 raft_consensus.cc:374] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:43.195333 11666 raft_consensus.cc:729] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efba900d3fac4e4dbf5f89860749f96c, State: Initialized, Role: FOLLOWER
I20260809 06:19:43.195984 11666 consensus_queue.cc:260] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.197329 11667 raft_consensus.cc:348] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.197877 11667 raft_consensus.cc:374] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:43.197992 11667 raft_consensus.cc:729] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589fed3fea814338b69691e8e33a08d7, State: Initialized, Role: FOLLOWER
I20260809 06:19:43.198766 11667 consensus_queue.cc:260] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.199359 11665 ts_tablet_manager.cc:1428] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260809 06:19:43.201856 11666 ts_tablet_manager.cc:1428] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Time spent starting tablet: real 0.018s user 0.003s sys 0.013s
I20260809 06:19:43.202243 11475 heartbeater.cc:503] Master 127.3.137.126:45079 was elected leader, sending a full tablet report...
I20260809 06:19:43.208484 11667 ts_tablet_manager.cc:1428] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Time spent starting tablet: real 0.023s user 0.019s sys 0.000s
I20260809 06:19:43.220376 11673 raft_consensus.cc:482] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:43.220660 11673 raft_consensus.cc:504] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.222735 11673 leader_election.cc:288] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815), efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
I20260809 06:19:43.230042 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.223074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:39045 (local address 127.3.137.67:45815)
0809 06:19:43.223304 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.223314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.224006 (+ 692us) server_negotiation.cc:407] Connection header received
0809 06:19:43.224245 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.224251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.224334 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.224431 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:43.225017 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.225986 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.228107 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.229074 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.229203 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.229257 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.229662 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.229788 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:19:43.230140 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.222948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45815 (local address 127.3.137.65:39045)
0809 06:19:43.223891 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.223907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.224100 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.224449 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.224455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.224479 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:43.224828 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.224841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.226127 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.226132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.227920 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.227934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.229292 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.229299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.229478 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.229518 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.229620 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.229759 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":940,"threads_started":1}
I20260809 06:19:43.230775 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.223441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:54377 (local address 127.3.137.66:37993)
0809 06:19:43.223693 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.223701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.224314 (+ 613us) server_negotiation.cc:407] Connection header received
0809 06:19:43.224528 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.224533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.224607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.224693 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:43.225474 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.226512 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.228666 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.229686 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.229834 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.229921 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.230388 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.230518 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 06:19:43.230873 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.223086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:37993 (local address 127.3.137.65:54377)
0809 06:19:43.224204 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:43.224219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:43.224372 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:43.224879 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:43.224885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:43.224913 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:43.225235 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.225267 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.226666 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.226670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.228520 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:43.228535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.229917 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.229925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:43.230111 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.230146 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:19:43.230352 (+ 206us) client_negotiation.cc:240] Negotiation successful
0809 06:19:43.230480 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":1,"thread_start_us":453,"threads_started":1}
I20260809 06:19:43.231253 11577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed94a14f41f04d7b8b51b698a8d9930b" candidate_uuid: "589fed3fea814338b69691e8e33a08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" is_pre_election: true
I20260809 06:19:43.231775 11577 raft_consensus.cc:2454] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589fed3fea814338b69691e8e33a08d7 in term 0.
I20260809 06:19:43.231835 11426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed94a14f41f04d7b8b51b698a8d9930b" candidate_uuid: "589fed3fea814338b69691e8e33a08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efba900d3fac4e4dbf5f89860749f96c" is_pre_election: true
I20260809 06:19:43.232381 11426 raft_consensus.cc:2454] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589fed3fea814338b69691e8e33a08d7 in term 0.
I20260809 06:19:43.232404 11201 leader_election.cc:302] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589fed3fea814338b69691e8e33a08d7, 90cb0dbd19c3490baa9657a7ca1b46ff; no voters:
I20260809 06:19:43.232774 11673 raft_consensus.cc:2781] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:43.232947 11673 raft_consensus.cc:482] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:43.233107 11673 raft_consensus.cc:3037] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Advancing to term 1
W20260809 06:19:43.234258 11323 tablet.cc:2369] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:43.235648 11673 raft_consensus.cc:504] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.236524 11673 leader_election.cc:288] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [CANDIDATE]: Term 1 election: Requested vote from peers 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815), efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993)
I20260809 06:19:43.236946 11577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed94a14f41f04d7b8b51b698a8d9930b" candidate_uuid: "589fed3fea814338b69691e8e33a08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
I20260809 06:19:43.237048 11426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed94a14f41f04d7b8b51b698a8d9930b" candidate_uuid: "589fed3fea814338b69691e8e33a08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efba900d3fac4e4dbf5f89860749f96c"
I20260809 06:19:43.237171 11577 raft_consensus.cc:3037] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:43.237323 11426 raft_consensus.cc:3037] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:43.240705 11577 raft_consensus.cc:2454] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589fed3fea814338b69691e8e33a08d7 in term 1.
I20260809 06:19:43.240705 11426 raft_consensus.cc:2454] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589fed3fea814338b69691e8e33a08d7 in term 1.
I20260809 06:19:43.241071 11201 leader_election.cc:302] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589fed3fea814338b69691e8e33a08d7, 90cb0dbd19c3490baa9657a7ca1b46ff; no voters:
I20260809 06:19:43.241362 11673 raft_consensus.cc:2781] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:43.241796 11673 raft_consensus.cc:686] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 1 LEADER]: Becoming Leader. State: Replica: 589fed3fea814338b69691e8e33a08d7, State: Running, Role: LEADER
I20260809 06:19:43.242338 11673 consensus_queue.cc:237] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } }
I20260809 06:19:43.248189 10254 catalog_manager.cc:5495] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 589fed3fea814338b69691e8e33a08d7 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "589fed3fea814338b69691e8e33a08d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } health_report { overall_health: UNKNOWN } } }
W20260809 06:19:43.283324 11630 tablet.cc:2369] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:43.293323 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.287776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.3.137.65:40567)
0809 06:19:43.288015 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.288021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.288042 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:43.288227 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.288231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.288331 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.288496 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:43.288909 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.289738 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.291116 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.291595 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.291694 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.292168 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:43.292782 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:43.292896 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.293031 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.293104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:19:43.302327 11426 raft_consensus.cc:1261] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Refusing update from remote peer 589fed3fea814338b69691e8e33a08d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:43.302328 11577 raft_consensus.cc:1261] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Refusing update from remote peer 589fed3fea814338b69691e8e33a08d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:43.303445 11677 consensus_queue.cc:1035] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:43.303830 11673 consensus_queue.cc:1035] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:43.316165 11680 mvcc.cc:205] Tried to move back new op lower bound from 7316506145994752000 to 7316506145764536320. Current Snapshot: MvccSnapshot[applied={T|T < 7316506145994752000}]
I20260809 06:19:43.336856 10254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:43.337131 10254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:43.344342 10254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efb59627dc9d4d75ab7f052e5bdbf070: Sending DeleteTablet for 3 replicas of tablet ed94a14f41f04d7b8b51b698a8d9930b
I20260809 06:19:43.345608 11246 tablet_service.cc:1514] Processing DeleteTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:43 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:50340
I20260809 06:19:43.345664 11557 tablet_service.cc:1514] Processing DeleteTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:43 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:37862
I20260809 06:19:43.345664 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet ed94a14f41f04d7b8b51b698a8d9930b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:43 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:59840
I20260809 06:19:43.346527 11691 tablet_replica.cc:330] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: stopping tablet replica
I20260809 06:19:43.347025 11691 raft_consensus.cc:2229] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:43.347764 11691 raft_consensus.cc:2258] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:43.349814 11691 ts_tablet_manager.cc:1905] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:43.357396 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10217
I20260809 06:19:43.358238 11692 tablet_replica.cc:330] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: stopping tablet replica
I20260809 06:19:43.358410 10284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:43.358659 11692 raft_consensus.cc:2229] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:43.359166 11692 raft_consensus.cc:2258] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:43.361261 11692 ts_tablet_manager.cc:1905] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:43.372287 11693 tablet_replica.cc:330] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: stopping tablet replica
I20260809 06:19:43.372917 11693 raft_consensus.cc:2229] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:43.375402 11693 raft_consensus.cc:2258] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:43.378937 11691 ts_tablet_manager.cc:1918] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:43.379437 11691 log.cc:1196] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/ed94a14f41f04d7b8b51b698a8d9930b
I20260809 06:19:43.380681 11691 ts_tablet_manager.cc:1939] T ed94a14f41f04d7b8b51b698a8d9930b P 589fed3fea814338b69691e8e33a08d7: Deleting consensus metadata
I20260809 06:19:43.399953 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.349217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.3.137.125:38263)
0809 06:19:43.349473 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.349482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.356112 (+ 6630us) server_negotiation.cc:407] Connection header received
0809 06:19:43.357163 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.357172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.357279 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.357372 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.373334 (+ 15962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.374313 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.377417 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.378490 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.378589 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.391877 (+ 13288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.391884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.391889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.391934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.396133 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.397158 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.397163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.397166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.397391 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.397918 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.397923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.397927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.398286 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.398545 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.398997 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.399239 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 06:19:43.404060 11692 ts_tablet_manager.cc:1918] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:43.404280 11692 log.cc:1196] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/ed94a14f41f04d7b8b51b698a8d9930b
I20260809 06:19:43.404860 11692 ts_tablet_manager.cc:1939] T ed94a14f41f04d7b8b51b698a8d9930b P efba900d3fac4e4dbf5f89860749f96c: Deleting consensus metadata
I20260809 06:19:43.415318 11693 ts_tablet_manager.cc:1905] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:43.417932 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.356151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.3.137.124:45459)
0809 06:19:43.356439 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.356447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.399908 (+ 43461us) server_negotiation.cc:407] Connection header received
0809 06:19:43.400181 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.400188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.400300 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.400381 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.402681 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.403821 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.405801 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.406469 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.406611 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.410652 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.410659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.410665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.410707 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.414427 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.415264 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.415270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.415273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.415414 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.415959 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.415964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.415967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.416366 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.416627 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.417223 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.417611 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:19:43.431371 11693 ts_tablet_manager.cc:1918] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:43.431562 11693 log.cc:1196] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/ed94a14f41f04d7b8b51b698a8d9930b
I20260809 06:19:43.432118 11693 ts_tablet_manager.cc:1939] T ed94a14f41f04d7b8b51b698a8d9930b P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting consensus metadata
I20260809 06:19:43.522578 10235 catalog_manager.cc:4841] TS efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993): tablet ed94a14f41f04d7b8b51b698a8d9930b (table test-table [id=847924f0bde443e18450eedd382c9add]) successfully deleted
I20260809 06:19:43.522953 10238 catalog_manager.cc:4841] TS 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815): tablet ed94a14f41f04d7b8b51b698a8d9930b (table test-table [id=847924f0bde443e18450eedd382c9add]) successfully deleted
I20260809 06:19:43.523228 10238 catalog_manager.cc:4841] TS 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567): tablet ed94a14f41f04d7b8b51b698a8d9930b (table test-table [id=847924f0bde443e18450eedd382c9add]) successfully deleted
I20260809 06:19:43.523710 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10217
I20260809 06:19:43.610067 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.574320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.3.137.125:38263)
0809 06:19:43.574533 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.574543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.585610 (+ 11067us) server_negotiation.cc:407] Connection header received
0809 06:19:43.585727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.585735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.585884 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.586006 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.588599 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.590143 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.592277 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.593084 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.593252 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.599030 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.599039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.599048 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.599100 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.603654 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.606826 (+ 3172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.606835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.606842 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.607056 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.607810 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.607818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.607823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.608269 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.608565 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.609299 (+ 734us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.609575 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:43.611186 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:43.585974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.3.137.124:45459)
0809 06:19:43.586160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:43.586166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:43.586799 (+ 633us) server_negotiation.cc:407] Connection header received
0809 06:19:43.587493 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:43.587502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:43.587642 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:43.587725 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:43.594201 (+ 6476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.595371 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.597638 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:43.598467 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:43.598590 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:43.602566 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:43.602574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:43.602579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:43.602622 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:43.606617 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.607442 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.607447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.607451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.607637 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:43.608063 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:43.608072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:43.608075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:43.608514 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:43.608735 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:43.610564 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0809 06:19:43.610826 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:19:44.324993 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.323225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:45949)
0809 06:19:44.324085 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.324441 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":440,"threads_started":1}
W20260809 06:19:44.325856 11475 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.332140 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.330376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:57833)
0809 06:19:44.331438 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.331750 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":758,"threads_started":1}
W20260809 06:19:44.333005 11320 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.334995 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.333216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:45027)
0809 06:19:44.334164 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.334432 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":521,"threads_started":1}
W20260809 06:19:44.335595 11628 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.844482 11706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader efb59627dc9d4d75ab7f052e5bdbf070)
I20260809 06:19:44.844854 11706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } }
I20260809 06:19:44.845258 11707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader efb59627dc9d4d75ab7f052e5bdbf070)
I20260809 06:19:44.845564 11707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } }
I20260809 06:19:44.846359 11706 leader_election.cc:288] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079), 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125:38263)
I20260809 06:19:44.846936 11707 leader_election.cc:288] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079), 24c31bc7f7b2417eaf823fd275d3c984 (127.3.137.124:45459)
I20260809 06:19:44.848802 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.846629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45610)
0809 06:19:44.847897 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.848202 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":1352,"threads_started":1}
W20260809 06:19:44.849381 11109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.851606 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.851147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45618)
0809 06:19:44.851358 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.851453 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:19:44.851884 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.847395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45616)
0809 06:19:44.850688 (+ 3293us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.851398 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3087,"thread_start_us":2308,"threads_started":1}
W20260809 06:19:44.851907 11109 leader_election.cc:334] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079): Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
W20260809 06:19:44.852332 10897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.860056 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.859354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45634)
0809 06:19:44.859641 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.859806 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260809 06:19:44.860306 10897 leader_election.cc:334] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079): Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.862880 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.846713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:38263 (local address 127.0.0.1:46902)
0809 06:19:44.848499 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.848534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:44.848813 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:44.852371 (+ 3558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:44.852376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:44.852418 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:44.853026 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.853055 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.856297 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.856313 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.859748 (+ 3435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.859766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.862007 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.862013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.862238 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.862281 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 06:19:44.862365 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 06:19:44.862492 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"thread_start_us":719,"threads_started":1}
I20260809 06:19:44.863335 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.846899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.3.137.125:38263)
0809 06:19:44.850091 (+ 3192us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:44.850100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:44.850132 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:19:44.850243 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:44.850258 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:44.851065 (+ 807us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:44.852358 (+ 1293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:44.853449 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.855992 (+ 2543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.859997 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.861802 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.861989 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.862310 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:44.862445 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 06:19:44.862693 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":2907,"thread_start_us":2386,"threads_started":1}
I20260809 06:19:44.864071 10922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c31bc7f7b2417eaf823fd275d3c984" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7105adb3d3484f50ba33155705b6b3e8" is_pre_election: true
I20260809 06:19:44.864459 10922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c31bc7f7b2417eaf823fd275d3c984 in term 1.
I20260809 06:19:44.865108 11108 leader_election.cc:302] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24c31bc7f7b2417eaf823fd275d3c984, 7105adb3d3484f50ba33155705b6b3e8; no voters: efb59627dc9d4d75ab7f052e5bdbf070
I20260809 06:19:44.865609 11706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 06:19:44.865871 11706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Starting leader election (detected failure of leader efb59627dc9d4d75ab7f052e5bdbf070)
I20260809 06:19:44.866052 11706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:19:44.868062 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.852993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:45459 (local address 127.0.0.1:40460)
0809 06:19:44.858144 (+ 5151us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.858183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:44.858412 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:44.859002 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:44.859010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:44.859065 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:44.859657 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.859679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.861410 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.861423 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.864497 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.864510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.866648 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.866662 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.867002 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.867060 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 06:19:44.867204 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 06:19:44.867408 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4883,"mutex_wait_us":1,"thread_start_us":4686,"threads_started":1}
I20260809 06:19:44.868134 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.850148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.3.137.124:45459)
0809 06:19:44.857648 (+ 7500us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:44.857662 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:44.858603 (+ 941us) server_negotiation.cc:407] Connection header received
0809 06:19:44.858688 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:44.858692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:44.858771 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:44.858985 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:44.859966 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.861230 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.864685 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.866377 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.866612 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.866813 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:44.867265 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 06:19:44.867505 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7293,"thread_start_us":7071,"threads_started":1}
I20260809 06:19:44.868980 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7105adb3d3484f50ba33155705b6b3e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "24c31bc7f7b2417eaf823fd275d3c984" is_pre_election: true
I20260809 06:19:44.870697 11706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } }
I20260809 06:19:44.871488 11134 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7105adb3d3484f50ba33155705b6b3e8 in current term 2: Already voted for candidate 24c31bc7f7b2417eaf823fd275d3c984 in this term.
I20260809 06:19:44.872069 11706 leader_election.cc:288] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 election: Requested vote from peers efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079), 7105adb3d3484f50ba33155705b6b3e8 (127.3.137.125:38263)
I20260809 06:19:44.872918 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.872097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45650)
0809 06:19:44.872500 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.872652 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260809 06:19:44.873052 10922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24c31bc7f7b2417eaf823fd275d3c984" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7105adb3d3484f50ba33155705b6b3e8"
I20260809 06:19:44.873200 10896 leader_election.cc:302] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7105adb3d3484f50ba33155705b6b3e8; no voters: 24c31bc7f7b2417eaf823fd275d3c984, efb59627dc9d4d75ab7f052e5bdbf070
I20260809 06:19:44.873449 10922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:19:44.875499 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.874942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45666)
0809 06:19:44.875248 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.875353 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260809 06:19:44.875801 11109 leader_election.cc:334] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079): Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
I20260809 06:19:44.877509 10922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c31bc7f7b2417eaf823fd275d3c984 in term 2.
I20260809 06:19:44.877744 11707 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:19:44.878175 11108 leader_election.cc:302] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24c31bc7f7b2417eaf823fd275d3c984, 7105adb3d3484f50ba33155705b6b3e8; no voters: efb59627dc9d4d75ab7f052e5bdbf070
I20260809 06:19:44.878647 11706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:19:44.879418 11706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [term 2 LEADER]: Becoming Leader. State: Replica: 24c31bc7f7b2417eaf823fd275d3c984, State: Running, Role: LEADER
I20260809 06:19:44.880033 11706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } }
I20260809 06:19:44.882623 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c31bc7f7b2417eaf823fd275d3c984. Latest consensus state: current_term: 2 leader_uuid: "24c31bc7f7b2417eaf823fd275d3c984" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:44.883308 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:44.884371 11719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:44.890308 11719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:44.891575 11719 catalog_manager.cc:1259] Loaded cluster ID: a0b918c1c940455382b3d40324d9b228
I20260809 06:19:44.891680 11719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:44.893293 11719 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:44.894435 11719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Loaded TSK: 0
I20260809 06:19:44.895721 11719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:19:44.936321 11122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:44.946650 11122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:44.952577 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.952031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45690)
0809 06:19:44.952259 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.952387 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 06:19:44.952776 10922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [term 2 FOLLOWER]: Refusing update from remote peer 24c31bc7f7b2417eaf823fd275d3c984: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 06:19:44.954149 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.953427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45698)
0809 06:19:44.953716 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.953887 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260809 06:19:44.953997 11717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 06:19:44.954519 11109 consensus_peers.cc:487] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 -> Peer efb59627dc9d4d75ab7f052e5bdbf070 (127.3.137.126:45079): Couldn't send request to peer efb59627dc9d4d75ab7f052e5bdbf070. Status: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 06:19:44.957870 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c31bc7f7b2417eaf823fd275d3c984. Latest consensus state: current_term: 2 leader_uuid: "24c31bc7f7b2417eaf823fd275d3c984" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:44.958462 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:44.960690 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "24c31bc7f7b2417eaf823fd275d3c984" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:44.961243 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:44.961853 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "24c31bc7f7b2417eaf823fd275d3c984" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "efb59627dc9d4d75ab7f052e5bdbf070" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45079 } } peers { permanent_uuid: "7105adb3d3484f50ba33155705b6b3e8" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 38263 } attrs { promote: false } } peers { permanent_uuid: "24c31bc7f7b2417eaf823fd275d3c984" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 45459 } attrs { promote: false } } }
I20260809 06:19:44.962266 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7105adb3d3484f50ba33155705b6b3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:19:44.986142 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.975907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:37993 (local address 127.0.0.1:59848)
0809 06:19:44.976088 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.976115 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:44.976231 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:44.977665 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:44.977671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:44.977712 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:44.978001 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.978032 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.980079 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.980097 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.982159 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.982177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.984715 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.984732 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.985205 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.985249 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 06:19:44.985493 (+ 244us) client_negotiation.cc:240] Negotiation successful
0809 06:19:44.985744 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:19:44.986910 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.976410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.3.137.66:37993)
0809 06:19:44.977163 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:44.977173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:44.977190 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:44.977271 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:44.977275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:44.977352 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:44.977520 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:44.978536 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.979838 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.982397 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.984371 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.984714 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:44.984962 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:44.985536 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 06:19:44.985767 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":2359,"threads_started":1}
I20260809 06:19:44.988497 11405 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f13b78f324f879a886684276c10c7 (DEFAULT_TABLE table=test-table [id=c64da6d73f054066b2fa3308044d23f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:44.989354 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f13b78f324f879a886684276c10c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:45.000962 11731 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Bootstrap starting.
I20260809 06:19:45.001927 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.977109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45815 (local address 127.0.0.1:37876)
0809 06:19:44.977308 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.977338 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:44.977513 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:44.991483 (+ 13970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:44.991489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:44.991551 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:44.992213 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.992253 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.994299 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.994311 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.997900 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.997925 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.000530 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.000548 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.000875 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.000943 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 06:19:45.001051 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:19:45.001439 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":47}
I20260809 06:19:45.003196 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.978420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.3.137.67:45815)
0809 06:19:44.990283 (+ 11863us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:44.990302 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:44.990338 (+ 36us) server_negotiation.cc:407] Connection header received
0809 06:19:44.990480 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:44.990489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:44.990673 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:44.991024 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:44.992492 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.994097 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.998307 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.000162 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.002433 (+ 2271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.002543 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.002606 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.002781 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11592,"thread_start_us":12915,"threads_started":1}
I20260809 06:19:45.003324 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.977560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40567 (local address 127.0.0.1:50360)
0809 06:19:44.978715 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:44.978765 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:44.978995 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:44.994048 (+ 15053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:44.994056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:44.994097 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:44.994616 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.994634 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.996600 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.996610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:44.999447 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:44.999470 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.001886 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.001918 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.002235 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.002301 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 06:19:45.002427 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 06:19:45.002683 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":5,"thread_start_us":1500,"threads_started":1}
I20260809 06:19:45.004542 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:44.977873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.3.137.65:40567)
0809 06:19:44.993414 (+ 15541us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:44.993447 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:44.993484 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:19:44.993585 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:44.993591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:44.993717 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:44.994860 (+ 1143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:44.995274 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:44.996367 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:44.999680 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.001380 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.003162 (+ 1782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.003825 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.003894 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.004097 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15281,"thread_start_us":15938,"threads_started":1}
I20260809 06:19:45.004487 11557 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f13b78f324f879a886684276c10c7 (DEFAULT_TABLE table=test-table [id=c64da6d73f054066b2fa3308044d23f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:45.004949 11731 tablet_bootstrap.cc:654] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:45.005371 11557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f13b78f324f879a886684276c10c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:45.005959 11246 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f13b78f324f879a886684276c10c7 (DEFAULT_TABLE table=test-table [id=c64da6d73f054066b2fa3308044d23f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:45.006516 11246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f13b78f324f879a886684276c10c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:45.013821 11731 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: No bootstrap required, opened a new log
I20260809 06:19:45.014165 11731 ts_tablet_manager.cc:1397] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260809 06:19:45.015667 11731 raft_consensus.cc:348] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.016085 11731 raft_consensus.cc:374] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:45.016280 11731 raft_consensus.cc:729] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efba900d3fac4e4dbf5f89860749f96c, State: Initialized, Role: FOLLOWER
I20260809 06:19:45.016723 11731 consensus_queue.cc:260] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.016986 11734 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Bootstrap starting.
I20260809 06:19:45.018710 11736 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Bootstrap starting.
I20260809 06:19:45.019966 11731 ts_tablet_manager.cc:1428] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260809 06:19:45.021679 11734 tablet_bootstrap.cc:654] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:45.026590 11474 heartbeater.cc:503] Master 127.3.137.124:45459 was elected leader, sending a full tablet report...
I20260809 06:19:45.027323 11736 tablet_bootstrap.cc:654] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:45.029362 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.021214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:34453)
0809 06:19:45.028564 (+ 7350us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.028890 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7138,"threads_started":1}
I20260809 06:19:45.031751 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.031018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:59711)
0809 06:19:45.031330 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.031490 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260809 06:19:45.034920 11736 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: No bootstrap required, opened a new log
I20260809 06:19:45.035277 11736 ts_tablet_manager.cc:1397] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260809 06:19:45.036671 11736 raft_consensus.cc:348] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.037009 11736 raft_consensus.cc:374] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:45.037222 11736 raft_consensus.cc:729] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cb0dbd19c3490baa9657a7ca1b46ff, State: Initialized, Role: FOLLOWER
I20260809 06:19:45.037751 11736 consensus_queue.cc:260] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.038630 11734 tablet_bootstrap.cc:492] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: No bootstrap required, opened a new log
I20260809 06:19:45.039009 11734 ts_tablet_manager.cc:1397] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20260809 06:19:45.039286 11736 ts_tablet_manager.cc:1428] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 06:19:45.040307 11734 raft_consensus.cc:348] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.040856 11734 raft_consensus.cc:374] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:45.041105 11734 raft_consensus.cc:729] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589fed3fea814338b69691e8e33a08d7, State: Initialized, Role: FOLLOWER
I20260809 06:19:45.041570 11734 consensus_queue.cc:260] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.043112 11627 heartbeater.cc:503] Master 127.3.137.124:45459 was elected leader, sending a full tablet report...
I20260809 06:19:45.045528 11734 ts_tablet_manager.cc:1428] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260809 06:19:45.047978 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.040177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:42905)
0809 06:19:45.047440 (+ 7263us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.047644 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7010,"threads_started":1}
I20260809 06:19:45.049909 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.047072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:39469)
0809 06:19:45.049223 (+ 2151us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.049554 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1939,"threads_started":1}
I20260809 06:19:45.051973 11319 heartbeater.cc:503] Master 127.3.137.124:45459 was elected leader, sending a full tablet report...
I20260809 06:19:45.053084 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.052394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:36821)
0809 06:19:45.052656 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.052841 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:19:45.055112 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.054346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:49713)
0809 06:19:45.054756 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.054877 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260809 06:19:45.087383 11735 raft_consensus.cc:482] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:45.087841 11735 raft_consensus.cc:504] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.090288 11735 leader_election.cc:288] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815), 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:45.104519 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.090733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:42539 (local address 127.3.137.67:45815)
0809 06:19:45.091130 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.091136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.091157 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:45.091462 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.091468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.091573 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.091718 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:45.092778 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.094277 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.100843 (+ 6566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.102638 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.102920 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.103101 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.103955 (+ 854us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.104149 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 06:19:45.104605 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.090640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45815 (local address 127.3.137.66:42539)
0809 06:19:45.090914 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.091022 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:45.091245 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:45.091744 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:45.091755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:45.091818 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:45.092462 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:45.092488 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.094469 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.094473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.100475 (+ 6002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:45.100509 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.103256 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.103273 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.103686 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.103748 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 06:19:45.103917 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 06:19:45.104149 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":234}
I20260809 06:19:45.105554 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.091319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:47983 (local address 127.3.137.65:40567)
0809 06:19:45.091652 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.091661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.092758 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 06:19:45.093005 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.093025 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.093293 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.093460 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:45.094037 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.095293 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.101038 (+ 5745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.103366 (+ 2328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.103684 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.103848 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.104783 (+ 935us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.105076 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 06:19:45.105523 11577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f13b78f324f879a886684276c10c7" candidate_uuid: "efba900d3fac4e4dbf5f89860749f96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" is_pre_election: true
I20260809 06:19:45.105948 11577 raft_consensus.cc:2454] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efba900d3fac4e4dbf5f89860749f96c in term 0.
I20260809 06:19:45.106834 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.091205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40567 (local address 127.3.137.66:47983)
0809 06:19:45.092578 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.092596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:45.092781 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:45.093472 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:45.093475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:45.093500 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:45.093858 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:45.093872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.095526 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.095534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.100518 (+ 4984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:45.100546 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.104107 (+ 3561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.104124 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:45.104464 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.104527 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 06:19:45.105829 (+ 1302us) client_negotiation.cc:240] Negotiation successful
0809 06:19:45.106032 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":213,"thread_start_us":855,"threads_started":1}
I20260809 06:19:45.106787 11360 leader_election.cc:302] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90cb0dbd19c3490baa9657a7ca1b46ff, efba900d3fac4e4dbf5f89860749f96c; no voters:
I20260809 06:19:45.107492 11735 raft_consensus.cc:2781] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:45.107795 11735 raft_consensus.cc:482] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:45.108021 11735 raft_consensus.cc:3037] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:45.108353 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f13b78f324f879a886684276c10c7" candidate_uuid: "efba900d3fac4e4dbf5f89860749f96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589fed3fea814338b69691e8e33a08d7" is_pre_election: true
I20260809 06:19:45.109120 11267 raft_consensus.cc:2454] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efba900d3fac4e4dbf5f89860749f96c in term 0.
I20260809 06:19:45.112747 11735 raft_consensus.cc:504] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.113649 11735 leader_election.cc:288] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [CANDIDATE]: Term 1 election: Requested vote from peers 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815), 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567)
I20260809 06:19:45.114315 11577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f13b78f324f879a886684276c10c7" candidate_uuid: "efba900d3fac4e4dbf5f89860749f96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff"
I20260809 06:19:45.114393 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f13b78f324f879a886684276c10c7" candidate_uuid: "efba900d3fac4e4dbf5f89860749f96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589fed3fea814338b69691e8e33a08d7"
I20260809 06:19:45.114821 11577 raft_consensus.cc:3037] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:45.114821 11267 raft_consensus.cc:3037] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:45.119824 11267 raft_consensus.cc:2454] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efba900d3fac4e4dbf5f89860749f96c in term 1.
I20260809 06:19:45.119824 11577 raft_consensus.cc:2454] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efba900d3fac4e4dbf5f89860749f96c in term 1.
I20260809 06:19:45.120517 11360 leader_election.cc:302] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589fed3fea814338b69691e8e33a08d7, efba900d3fac4e4dbf5f89860749f96c; no voters:
I20260809 06:19:45.120882 11735 raft_consensus.cc:2781] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:45.121762 11735 raft_consensus.cc:686] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 1 LEADER]: Becoming Leader. State: Replica: efba900d3fac4e4dbf5f89860749f96c, State: Running, Role: LEADER
I20260809 06:19:45.122578 11735 consensus_queue.cc:237] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } }
I20260809 06:19:45.124199 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.123214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:36669)
0809 06:19:45.123738 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.123930 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20260809 06:19:45.128435 11122 catalog_manager.cc:5495] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c reported cstate change: term changed from 0 to 1, leader changed from <none> to efba900d3fac4e4dbf5f89860749f96c (127.3.137.66). New cstate: current_term: 1 leader_uuid: "efba900d3fac4e4dbf5f89860749f96c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efba900d3fac4e4dbf5f89860749f96c" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 37993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 } health_report { overall_health: UNKNOWN } } }
W20260809 06:19:45.139149 11478 tablet.cc:2369] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:45.171980 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.163839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.3.137.66:37993)
0809 06:19:45.164216 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.164225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.164254 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:19:45.164391 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.164394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.164496 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.164712 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:45.165293 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.166326 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.168535 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.169219 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.169424 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.170117 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:45.170894 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:45.171255 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.171415 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.171562 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 06:19:45.178606 11577 raft_consensus.cc:1261] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Refusing update from remote peer efba900d3fac4e4dbf5f89860749f96c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:45.179360 11267 raft_consensus.cc:1261] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Refusing update from remote peer efba900d3fac4e4dbf5f89860749f96c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:45.180019 11735 consensus_queue.cc:1035] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90cb0dbd19c3490baa9657a7ca1b46ff" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:45.180588 11746 consensus_queue.cc:1035] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "589fed3fea814338b69691e8e33a08d7" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:45.185837 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.184073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:43053)
0809 06:19:45.185373 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.185522 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1092}
I20260809 06:19:45.188127 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.186752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:45953)
0809 06:19:45.187245 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.187559 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260809 06:19:45.193010 11749 mvcc.cc:205] Tried to move back new op lower bound from 7316506153680752640 to 7316506153469980672. Current Snapshot: MvccSnapshot[applied={T|T < 7316506153680752640}]
I20260809 06:19:45.197167 11750 mvcc.cc:205] Tried to move back new op lower bound from 7316506153680752640 to 7316506153469980672. Current Snapshot: MvccSnapshot[applied={T|T < 7316506153680752640}]
I20260809 06:19:45.198102 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.197207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:43879)
0809 06:19:45.197593 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.197772 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260809 06:19:45.200639 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.199357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.67:56837)
0809 06:19:45.200134 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.200299 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622}
I20260809 06:19:45.203841 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.203028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.66:49753)
0809 06:19:45.203362 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.203499 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260809 06:19:45.208911 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.201092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.3.137.65:36829)
0809 06:19:45.208408 (+ 7316us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.208636 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7063}
I20260809 06:19:45.232903 11122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:45.233273 11122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:45.244027 11122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24c31bc7f7b2417eaf823fd275d3c984: Sending DeleteTablet for 3 replicas of tablet ca8f13b78f324f879a886684276c10c7
I20260809 06:19:45.246259 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f13b78f324f879a886684276c10c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:45 UTC) from {username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:59848
I20260809 06:19:45.246445 11557 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f13b78f324f879a886684276c10c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:45 UTC) from {username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:37876
I20260809 06:19:45.246672 11246 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f13b78f324f879a886684276c10c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:45 UTC) from {username='oryx', principal='oryx/127.3.137.124@KRBTEST.COM'} at 127.0.0.1:50360
I20260809 06:19:45.257687 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 11093
I20260809 06:19:45.268980 11762 tablet_replica.cc:330] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: stopping tablet replica
I20260809 06:19:45.269084 11760 tablet_replica.cc:330] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: stopping tablet replica
I20260809 06:19:45.269441 11762 raft_consensus.cc:2229] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:45.269476 11760 raft_consensus.cc:2229] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:45.270152 11762 raft_consensus.cc:2258] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:45.270874 11760 raft_consensus.cc:2258] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:45.273738 11762 ts_tablet_manager.cc:1905] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:45.275146 11760 ts_tablet_manager.cc:1905] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:45.281389 11761 tablet_replica.cc:330] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: stopping tablet replica
I20260809 06:19:45.282150 11761 raft_consensus.cc:2229] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:45.282883 11761 raft_consensus.cc:2258] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:45.287837 11761 ts_tablet_manager.cc:1905] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:45.289398 11760 ts_tablet_manager.cc:1918] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:45.289677 11760 log.cc:1196] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/ca8f13b78f324f879a886684276c10c7
I20260809 06:19:45.290702 11760 ts_tablet_manager.cc:1939] T ca8f13b78f324f879a886684276c10c7 P efba900d3fac4e4dbf5f89860749f96c: Deleting consensus metadata
I20260809 06:19:45.292589 11762 ts_tablet_manager.cc:1918] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:45.292913 11762 log.cc:1196] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/ca8f13b78f324f879a886684276c10c7
I20260809 06:19:45.293123 11106 catalog_manager.cc:4841] TS efba900d3fac4e4dbf5f89860749f96c (127.3.137.66:37993): tablet ca8f13b78f324f879a886684276c10c7 (table test-table [id=c64da6d73f054066b2fa3308044d23f3]) successfully deleted
I20260809 06:19:45.293861 11762 ts_tablet_manager.cc:1939] T ca8f13b78f324f879a886684276c10c7 P 589fed3fea814338b69691e8e33a08d7: Deleting consensus metadata
I20260809 06:19:45.295698 11109 catalog_manager.cc:4841] TS 589fed3fea814338b69691e8e33a08d7 (127.3.137.65:40567): tablet ca8f13b78f324f879a886684276c10c7 (table test-table [id=c64da6d73f054066b2fa3308044d23f3]) successfully deleted
I20260809 06:19:45.301538 11761 ts_tablet_manager.cc:1918] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:45.301750 11761 log.cc:1196] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/ca8f13b78f324f879a886684276c10c7
I20260809 06:19:45.302320 11761 ts_tablet_manager.cc:1939] T ca8f13b78f324f879a886684276c10c7 P 90cb0dbd19c3490baa9657a7ca1b46ff: Deleting consensus metadata
I20260809 06:19:45.304134 11109 catalog_manager.cc:4841] TS 90cb0dbd19c3490baa9657a7ca1b46ff (127.3.137.67:45815): tablet ca8f13b78f324f879a886684276c10c7 (table test-table [id=c64da6d73f054066b2fa3308044d23f3]) successfully deleted
I20260809 06:19:45.308338 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.259746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.3.137.65:40567)
0809 06:19:45.260118 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.260137 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.267886 (+ 7749us) server_negotiation.cc:407] Connection header received
0809 06:19:45.268332 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.268351 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.268643 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.268871 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:45.274822 (+ 5951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.276815 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.283565 (+ 6750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.284907 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.285136 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.294382 (+ 9246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:45.294424 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:45.294446 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:45.294515 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:45.302580 (+ 8065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.304150 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.304172 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.304177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.304554 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.305293 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.305301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.305315 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.306243 (+ 928us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:45.306818 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.307236 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.307661 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 06:19:45.309425 11313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:45.340832 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.340110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45712)
0809 06:19:45.340467 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.340608 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 06:19:45.342779 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.342045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45716)
0809 06:19:45.342386 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.342536 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260809 06:19:45.631076 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 11093
I20260809 06:19:45.675474 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 11341
I20260809 06:19:45.708593 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.676671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.3.137.66:37993)
0809 06:19:45.677674 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.677681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.677711 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:19:45.677819 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.677829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.677943 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.678155 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:45.680703 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.682255 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.684839 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.686800 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.687132 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.692494 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:45.692539 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:45.692583 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:45.692750 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:45.701810 (+ 9060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.703103 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.703127 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.703132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.703666 (+ 534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.704493 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.704526 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.704557 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.705969 (+ 1412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:45.706685 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.707187 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.707667 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":532,"threads_started":1}
I20260809 06:19:45.710166 11469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:45.786733 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.786119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45730)
0809 06:19:45.786339 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.786477 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 06:19:45.788477 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.787840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45079 (local address 127.0.0.1:45734)
0809 06:19:45.788088 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:45.788242 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 06:19:45.926432 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 11341
I20260809 06:19:45.956472 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 11489
I20260809 06:19:45.978775 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:45.957687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.3.137.67:45815)
0809 06:19:45.958430 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:45.958438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:45.958463 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:19:45.958557 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:45.958561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:45.958707 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:45.958856 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:45.961346 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.962739 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.964910 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:45.965688 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:45.965838 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:45.970194 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:45.970218 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:45.970243 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:45.970343 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:45.974889 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.976024 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.976040 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.976042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.976294 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:45.976773 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:45.976776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:45.976778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:45.977316 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:45.977685 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:45.978093 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 06:19:45.978358 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":378,"threads_started":1}
I20260809 06:19:45.979967 11621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:46.095057 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 11489
I20260809 06:19:46.116340 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10760
I20260809 06:19:46.117447 10945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:46.240119 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10760
I20260809 06:19:46.265946 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 10872
I20260809 06:19:46.267159 11157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:46.401140 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 10872
2026-08-09T06:19:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10213 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:46 dist-test-slave-2w3w krb5kdc[11777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:46 dist-test-slave-2w3w krb5kdc[11777](info): set up 2 sockets
Aug 09 06:19:46 dist-test-slave-2w3w krb5kdc[11777](info): commencing operation
krb5kdc: starting...
W20260809 06:19:47.555658 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:47 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab.
Entry for principal oryx/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab.
I20260809 06:19:47.802557 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39717
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab
--principal=oryx/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:48.142576 11792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:48.142864 11792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:48.143002 11792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:48.150010 11792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:48.150107 11792 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:19:48.150148 11792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:48.150210 11792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:48.150261 11792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:48.150315 11792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:48.159890 11792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.126.keytab
--principal=oryx/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:48.161046 11792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:48.162261 11792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:48.174086 11797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:48.174149 11798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:48.174324 11800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:48.175316 11792 server_base.cc:1034] running on GCE node
I20260809 06:19:48.176143 11792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:48.177490 11792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:48.178742 11792 hybrid_clock.cc:648] HybridClock initialized: now 1786256388178674 us; error 42 us; skew 500 ppm
I20260809 06:19:48.179134 11792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:48 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256388, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:48.182258 11792 init.cc:377] Logged in from keytab as oryx/127.3.137.126@KRBTEST.COM (short username oryx)
I20260809 06:19:48.183800 11792 webserver.cc:460] Webserver started at http://127.3.137.126:37109/ using document root <none> and password file <none>
I20260809 06:19:48.184491 11792 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:48.184623 11792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:48.184967 11792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:48.187922 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "fff955601e204984a77be3cfc319f247"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "db5a550ed5a899112ebf379b3b301729"
server_key_iv: "6c98487d33dad9334fa9f268a845bb20"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.188714 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "fff955601e204984a77be3cfc319f247"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "db5a550ed5a899112ebf379b3b301729"
server_key_iv: "6c98487d33dad9334fa9f268a845bb20"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.193742 11792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:19:48.197141 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:48.198441 11792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:19:48.198634 11792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "fff955601e204984a77be3cfc319f247"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "db5a550ed5a899112ebf379b3b301729"
server_key_iv: "6c98487d33dad9334fa9f268a845bb20"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.198874 11792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:48.215960 11792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:48.219913 11792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:48.220178 11792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:48.242632 11792 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:39717
I20260809 06:19:48.242652 11875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:39717 every 8 connection(s)
I20260809 06:19:48.244110 11792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:48.247417 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 11792
I20260809 06:19:48.247675 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:48.248801 3621 external_mini_cluster.cc:1428] Setting key f1707f24ff82b33b04951db1111a3d03
I20260809 06:19:48.249069 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:48.264351 11876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Bootstrap starting.
Aug 09 06:19:48 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:48.267788 11876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:48.269356 11876 log.cc:824] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:48.273389 11876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: No bootstrap required, opened a new log
I20260809 06:19:48.275398 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:48.251179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.3.137.126:39717)
0809 06:19:48.252045 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:48.252055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:48.252101 (+ 46us) server_negotiation.cc:407] Connection header received
0809 06:19:48.254102 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:48.254124 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:48.254519 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:48.254860 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:48.256754 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.258093 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.259980 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.260697 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.260839 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:48.266120 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:48.266128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:48.266142 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:48.266185 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:48.270183 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.271205 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.271217 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.271225 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.271448 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.271848 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.271854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.271857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.272679 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:48.273005 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:48.273347 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 06:19:48.273822 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":411,"threads_started":1}
I20260809 06:19:48.283102 11876 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } }
I20260809 06:19:48.283516 11876 raft_consensus.cc:374] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:48.283643 11876 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff955601e204984a77be3cfc319f247, State: Initialized, Role: FOLLOWER
I20260809 06:19:48.284284 11876 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } }
I20260809 06:19:48.284511 11876 raft_consensus.cc:388] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:48.284611 11876 raft_consensus.cc:482] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:48.284744 11876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:48.286942 11876 raft_consensus.cc:504] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } }
I20260809 06:19:48.287497 11876 leader_election.cc:302] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff955601e204984a77be3cfc319f247; no voters:
I20260809 06:19:48.288166 11876 leader_election.cc:288] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:48.288303 11881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:48.289435 11881 raft_consensus.cc:686] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [term 1 LEADER]: Becoming Leader. State: Replica: fff955601e204984a77be3cfc319f247, State: Running, Role: LEADER
I20260809 06:19:48.290081 11881 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } }
I20260809 06:19:48.290616 11876 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:48.293270 11882 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff955601e204984a77be3cfc319f247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } } }
I20260809 06:19:48.293756 11882 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:48.295965 11883 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff955601e204984a77be3cfc319f247. Latest consensus state: current_term: 1 leader_uuid: "fff955601e204984a77be3cfc319f247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff955601e204984a77be3cfc319f247" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39717 } } }
I20260809 06:19:48.296270 11883 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:48.297282 11887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:48.304677 11887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:48.313613 11887 catalog_manager.cc:1347] Generated new cluster ID: 18f95143319c42629bce1d0fa827e03b
I20260809 06:19:48.313725 11887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:48.324172 11887 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:48.325412 11887 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:48.342459 11887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Generated new TSK 0
I20260809 06:19:48.343504 11887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
I20260809 06:19:48.407303 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:48.755599 11904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:48.755898 11904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:48.755998 11904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:48.763139 11904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:48.763257 11904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:48.763350 11904 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:48.763432 11904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:48.773931 11904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:48.775352 11904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:48.776813 11904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:48.791074 11909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:48.791136 11910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:48.791157 11912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:48.792291 11904 server_base.cc:1034] running on GCE node
I20260809 06:19:48.792901 11904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:48.793816 11904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:48.795073 11904 hybrid_clock.cc:648] HybridClock initialized: now 1786256388794999 us; error 49 us; skew 500 ppm
I20260809 06:19:48.795388 11904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:48 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256388, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:48.798774 11904 init.cc:377] Logged in from keytab as oryx/127.3.137.65@KRBTEST.COM (short username oryx)
I20260809 06:19:48.800272 11904 webserver.cc:460] Webserver started at http://127.3.137.65:34887/ using document root <none> and password file <none>
I20260809 06:19:48.800951 11904 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:48.801082 11904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:48.801427 11904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:48.804659 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "fd3b1eb34c20483993c4cc5dedc58de9"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "831d206e049706a9be699f95fc5ecaec"
server_key_iv: "abf0711103e63829183b76de8788ef06"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.805457 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "fd3b1eb34c20483993c4cc5dedc58de9"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "831d206e049706a9be699f95fc5ecaec"
server_key_iv: "abf0711103e63829183b76de8788ef06"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.810449 11904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:19:48.813875 11919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:48.815212 11904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:48.815395 11904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "fd3b1eb34c20483993c4cc5dedc58de9"
format_stamp: "Formatted at 2026-08-09 06:19:48 on dist-test-slave-2w3w"
server_key: "831d206e049706a9be699f95fc5ecaec"
server_key_iv: "abf0711103e63829183b76de8788ef06"
server_key_version: "encryptionkey@0"
I20260809 06:19:48.815621 11904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:48.826057 11904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:48.830845 11904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:48.831310 11904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:48.834163 11904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:48.834280 11904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:48.834432 11904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:48.834507 11904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:48 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256388, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:48.859175 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:48.835034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:43791 (local address 127.3.137.126:39717)
0809 06:19:48.835546 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:48.835551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:48.836686 (+ 1135us) server_negotiation.cc:407] Connection header received
0809 06:19:48.838928 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:48.838933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:48.839054 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:48.839186 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:48.841666 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.843121 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.845870 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.846812 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.847012 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:48.852022 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:48.852027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:48.852030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:48.852063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:48.855242 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.856488 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.856496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.856500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.856667 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.857267 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.857273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.857276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.857611 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:48.857765 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:48.858542 (+ 777us) server_negotiation.cc:299] Negotiation successful
0809 06:19:48.858768 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":282,"threads_started":1}
I20260809 06:19:48.860220 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:48.835677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.65:43791)
0809 06:19:48.836517 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:48.836555 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:48.838634 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:48.839658 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:48.839669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:48.840473 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:48.841422 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:48.841451 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.843367 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.843377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:48.845668 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:48.845692 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.847287 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.847307 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:48.847613 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:48.848374 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:48.848416 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:48.851776 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:48.855441 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:48.855450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:48.855472 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:48.856325 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:48.856818 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:48.856825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:48.856833 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:48.857090 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:48.857762 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:48.857771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:48.858165 (+ 394us) client_negotiation.cc:769] Sending connection context
0809 06:19:48.858442 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 06:19:48.859023 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260809 06:19:48.883723 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:48.870058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:57791 (local address 127.3.137.126:39717)
0809 06:19:48.870221 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:48.870226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:48.870876 (+ 650us) server_negotiation.cc:407] Connection header received
0809 06:19:48.871128 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:48.871132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:48.871195 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:48.871299 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:48.872607 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.873310 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.874627 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.875115 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.875248 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:48.877177 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:48.877186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:48.877193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:48.877240 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:48.880669 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.881371 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.881379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.881384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.881517 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:48.881906 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:48.881914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:48.881918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:48.882301 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:48.882501 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:48.883137 (+ 636us) server_negotiation.cc:299] Negotiation successful
0809 06:19:48.883364 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:19:48.883792 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:48.870131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.65:57791)
0809 06:19:48.870752 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:48.870772 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:48.870953 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:48.871336 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:48.871341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:48.871954 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:48.872448 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:48.872459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.873448 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.873453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:48.874490 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:48.874501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:48.875358 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:48.875368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:48.875528 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:48.876232 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:48.876249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:48.876980 (+ 731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:48.880804 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:48.880810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:48.880814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:48.881248 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:48.881629 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:48.881635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:48.881638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:48.881778 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:48.882455 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:48.882463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:48.882734 (+ 271us) client_negotiation.cc:769] Sending connection context
0809 06:19:48.882921 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 06:19:48.883336 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
I20260809 06:19:48.894501 11904 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:39019
I20260809 06:19:48.894531 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:39019 every 8 connection(s)
I20260809 06:19:48.895781 11904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:48.898005 12054 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39717
I20260809 06:19:48.898269 12054 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:48.899016 12054 heartbeater.cc:511] Master 127.3.137.126:39717 requested a full tablet report, sending...
I20260809 06:19:48.900935 11825 ts_manager.cc:194] Registered new tserver with Master: fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019)
I20260809 06:19:48.902346 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 11904
I20260809 06:19:48.902540 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:48.903028 11825 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:43791
I20260809 06:19:48.903508 3621 external_mini_cluster.cc:1428] Setting key a9370a442ebd2c839443b5bfd674e0c6
WARNING: no policy specified for oryx/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
I20260809 06:19:48.961936 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:49.314380 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:49.314692 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:49.314878 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:49.322546 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:49.322708 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:49.322853 12059 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:49.323040 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:49.333012 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:49.334309 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:49.335691 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:49.348331 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:49.348753 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:49.348779 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:49.349232 12059 server_base.cc:1034] running on GCE node
I20260809 06:19:49.349864 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:49.350778 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:49.352008 12059 hybrid_clock.cc:648] HybridClock initialized: now 1786256389351966 us; error 71 us; skew 500 ppm
I20260809 06:19:49.352311 12059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:49 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256389, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:49.355664 12059 init.cc:377] Logged in from keytab as oryx/127.3.137.66@KRBTEST.COM (short username oryx)
I20260809 06:19:49.357075 12059 webserver.cc:460] Webserver started at http://127.3.137.66:33379/ using document root <none> and password file <none>
I20260809 06:19:49.357766 12059 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:49.357903 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:49.358251 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:49.361219 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "4caa226cb74d41daa93c098cb1eaeb41"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "2b9387a3bd657a70abcb3c3e92e21514"
server_key_iv: "6bec73a6eec613de396d03af897af67e"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.362099 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "4caa226cb74d41daa93c098cb1eaeb41"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "2b9387a3bd657a70abcb3c3e92e21514"
server_key_iv: "6bec73a6eec613de396d03af897af67e"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.367377 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 06:19:49.370772 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:49.372044 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:49.372220 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "4caa226cb74d41daa93c098cb1eaeb41"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "2b9387a3bd657a70abcb3c3e92e21514"
server_key_iv: "6bec73a6eec613de396d03af897af67e"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.372450 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:49.394767 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:49.398731 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:49.399078 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:49.401803 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:49.401930 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:49.402104 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:49.402181 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:49 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256389, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:49.426273 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.402326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58995 (local address 127.3.137.126:39717)
0809 06:19:49.403340 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:49.403344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:49.404238 (+ 894us) server_negotiation.cc:407] Connection header received
0809 06:19:49.406287 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:49.406293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:49.406390 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:49.406549 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:49.408766 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.410170 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.412861 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.413294 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.413464 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.418597 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:49.418602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:49.418605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:49.418640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:49.422424 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.423517 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.423524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.423530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.423701 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.424430 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.424443 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.424446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.424746 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:49.424922 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:49.425637 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 06:19:49.425884 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":816,"threads_started":1}
I20260809 06:19:49.427373 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.403261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.66:58995)
0809 06:19:49.404089 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:49.404123 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:49.405950 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:49.406819 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:49.406830 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:49.407634 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:49.408460 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.408483 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.410366 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.410372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.412666 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.412682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.413801 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.413807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.414084 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.414910 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:49.414979 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:49.418357 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:49.422605 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.422614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.422630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.423377 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.423841 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.423852 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.423859 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.424264 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.424890 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:49.424898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:49.425268 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 06:19:49.425540 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 06:19:49.426088 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":397,"threads_started":1}
I20260809 06:19:49.449713 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.433948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.66:41463)
0809 06:19:49.434542 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:49.434561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:49.434727 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:49.435105 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:49.435109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:49.435928 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:49.436460 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.436479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.437828 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.437836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.439532 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.439550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.440750 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.440757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.440884 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.441590 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:49.441607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:49.442393 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:49.446741 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.446750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.446756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.447336 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.447816 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.447821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.447824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.447989 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.448820 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:49.448831 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:49.449008 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 06:19:49.449122 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 06:19:49.449346 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":342,"threads_started":1}
I20260809 06:19:49.450074 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.433555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:41463 (local address 127.3.137.126:39717)
0809 06:19:49.433726 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:49.433730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:49.434648 (+ 918us) server_negotiation.cc:407] Connection header received
0809 06:19:49.434865 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:49.434869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:49.434927 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:49.435073 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:49.436675 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.437603 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.439716 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.440532 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.440693 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.442609 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:49.442619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:49.442627 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:49.442674 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:49.446590 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.447516 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.447523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.447529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.447686 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.448129 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.448136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.448140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.448680 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:49.448849 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:49.449292 (+ 443us) server_negotiation.cc:299] Negotiation successful
0809 06:19:49.449616 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:19:49.456620 12059 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40399
I20260809 06:19:49.456661 12206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40399 every 8 connection(s)
I20260809 06:19:49.457911 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:49.460036 12207 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39717
I20260809 06:19:49.460260 12207 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:49.460863 12207 heartbeater.cc:511] Master 127.3.137.126:39717 requested a full tablet report, sending...
I20260809 06:19:49.462112 11825 ts_manager.cc:194] Registered new tserver with Master: 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399)
I20260809 06:19:49.462893 11825 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:58995
I20260809 06:19:49.468004 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12059
I20260809 06:19:49.468187 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:49.468667 3621 external_mini_cluster.cc:1428] Setting key 01b9ad89974f505a81e11614b8c83f3e
WARNING: no policy specified for oryx/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
I20260809 06:19:49.522787 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39717
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:49.860597 12211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:49.860915 12211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:49.861093 12211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:49.868474 12211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:49.868628 12211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:49.868770 12211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:19:49.868911 12211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:49.879426 12211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.137.126:39717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:49.880664 12211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:49.882005 12211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:49.895190 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:49.895218 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:49.896198 12211 server_base.cc:1034] running on GCE node
W20260809 06:19:49.896414 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:49.897246 12211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:49.898401 12211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:49.899660 12211 hybrid_clock.cc:648] HybridClock initialized: now 1786256389899570 us; error 49 us; skew 500 ppm
I20260809 06:19:49.900007 12211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:19:49.906363 12054 heartbeater.cc:503] Master 127.3.137.126:39717 was elected leader, sending a full tablet report...
Aug 09 06:19:49 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256389, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:49.909863 12211 init.cc:377] Logged in from keytab as oryx/127.3.137.67@KRBTEST.COM (short username oryx)
I20260809 06:19:49.911408 12211 webserver.cc:460] Webserver started at http://127.3.137.67:34935/ using document root <none> and password file <none>
I20260809 06:19:49.912129 12211 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:49.912245 12211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:49.912590 12211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:49.915659 12211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "daaa3eae3ccc4105b3551349d7e4e336"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "7538c312d8888ec79e64eb7e6e46150a"
server_key_iv: "ed65197fa2722bf8126d6d919861de10"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.916461 12211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "daaa3eae3ccc4105b3551349d7e4e336"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "7538c312d8888ec79e64eb7e6e46150a"
server_key_iv: "ed65197fa2722bf8126d6d919861de10"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.922055 12211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:19:49.925441 12226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:49.926860 12211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:19:49.927073 12211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "daaa3eae3ccc4105b3551349d7e4e336"
format_stamp: "Formatted at 2026-08-09 06:19:49 on dist-test-slave-2w3w"
server_key: "7538c312d8888ec79e64eb7e6e46150a"
server_key_iv: "ed65197fa2722bf8126d6d919861de10"
server_key_version: "encryptionkey@0"
I20260809 06:19:49.927356 12211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:49.944866 12211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:49.949005 12211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:49.949285 12211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:49.952140 12211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:49.952284 12211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:49.952471 12211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:49.952553 12211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:19:49 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256389, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:49.976722 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.952540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:54001 (local address 127.3.137.126:39717)
0809 06:19:49.953543 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:49.953552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:49.954764 (+ 1212us) server_negotiation.cc:407] Connection header received
0809 06:19:49.956730 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:49.956739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:49.956851 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:49.956991 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:49.959176 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.960430 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.963000 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.963603 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.963751 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.968834 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:49.968840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:49.968843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:49.968878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:49.972523 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.973766 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.973771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.973773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.973907 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.974524 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.974527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.974529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:49.974819 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:49.974930 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:49.976165 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0809 06:19:49.976431 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":728,"threads_started":1}
I20260809 06:19:49.978890 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.953334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.67:54001)
0809 06:19:49.954252 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:49.954307 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:49.956430 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:49.957268 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:49.957278 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:49.958065 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:49.958935 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.958996 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.960623 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.960631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.962812 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.962828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.963851 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.963858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.964092 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.964917 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:49.964955 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:49.968588 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:49.972749 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.972759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.972780 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.973611 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.974031 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.974039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.974045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.974397 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:49.974952 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:49.974993 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:49.975589 (+ 596us) client_negotiation.cc:769] Sending connection context
0809 06:19:49.975982 (+ 393us) client_negotiation.cc:240] Negotiation successful
0809 06:19:49.976758 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260809 06:19:50.002123 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.985444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:49451 (local address 127.3.137.126:39717)
0809 06:19:49.985668 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:49.985676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:49.986415 (+ 739us) server_negotiation.cc:407] Connection header received
0809 06:19:49.986620 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:49.986625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:49.986725 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:49.986809 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:49.988434 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.989172 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.991895 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.992544 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.992698 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.995442 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:49.995447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:49.995450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:49.995483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:49.998856 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:49.999904 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:49.999910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:49.999913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:50.000118 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:50.000623 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:50.000627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:50.000629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:50.000901 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:50.001029 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:50.001655 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 06:19:50.001852 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:19:50.002650 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:49.985468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.3.137.67:49451)
0809 06:19:49.986245 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:49.986270 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:49.986481 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:49.986876 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:49.986884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:49.987706 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:49.988243 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.988261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.989339 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.989348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.991710 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:49.991732 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:49.992778 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:49.992787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:49.993033 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:49.993765 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:49.993795 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:49.995222 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:49.999056 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:49.999062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:49.999066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:49.999781 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:50.000262 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:50.000270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:50.000287 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:50.000490 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:50.001022 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:50.001030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:50.001307 (+ 277us) client_negotiation.cc:769] Sending connection context
0809 06:19:50.001507 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 06:19:50.001969 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":395,"threads_started":1}
I20260809 06:19:50.013141 12211 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33331
I20260809 06:19:50.013161 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33331 every 8 connection(s)
I20260809 06:19:50.014374 12211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:50.016588 12357 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39717
I20260809 06:19:50.016821 12357 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:50.017495 12357 heartbeater.cc:511] Master 127.3.137.126:39717 requested a full tablet report, sending...
I20260809 06:19:50.018010 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12211
I20260809 06:19:50.018189 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:50.018751 3621 external_mini_cluster.cc:1428] Setting key 5f12e938f2a2a4edb44ec154446c3f20
I20260809 06:19:50.018908 11825 ts_manager.cc:194] Registered new tserver with Master: daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:50.019959 11825 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:54001
I20260809 06:19:50.022215 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:19:50 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:19:50 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:19:50 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:19:50.465209 12207 heartbeater.cc:503] Master 127.3.137.126:39717 was elected leader, sending a full tablet report...
I20260809 06:19:51.022698 12357 heartbeater.cc:503] Master 127.3.137.126:39717 was elected leader, sending a full tablet report...
W20260809 06:19:51.151280 12365 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.077s user 0.000s sys 0.019s
W20260809 06:19:51.152601 12234 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:50.068537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.3.137.126:39717)
0809 06:19:50.068715 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:50.068720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:50.068992 (+ 272us) server_negotiation.cc:407] Connection header received
0809 06:19:50.069161 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:50.069165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:50.069233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:50.069341 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:50.070679 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:50.071494 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:50.072741 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:50.073136 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:50.073229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.152305 (+1079076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41808: BlockingRecv error: recv got EOF from 127.0.0.1:41808 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 06:19:51 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:51.170109 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.156453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.3.137.126:39717)
0809 06:19:51.156606 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.156610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.156936 (+ 326us) server_negotiation.cc:407] Connection header received
0809 06:19:51.157100 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.157103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.157167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.157257 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:51.158470 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.159253 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.160538 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.160978 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.161095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.165264 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:51.165269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:51.165273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:51.165305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:51.167866 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:51.168460 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:51.168463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:51.168466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:51.168585 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:51.168890 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:51.168893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:51.168895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:51.169144 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:51.169333 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.169595 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.169763 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:19:51.174216 11825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:51.181344 11825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:51.216658 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.193244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33331 (local address 127.0.0.1:56836)
0809 06:19:51.194155 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.194214 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.194458 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.208392 (+ 13934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.208399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.208428 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:51.209169 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.209183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.210898 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.210903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.213647 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.213662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.215620 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.215627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.215845 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.215898 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.216003 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.216192 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":27,"spinlock_wait_cycles":319872,"thread_start_us":402,"threads_started":1}
I20260809 06:19:51.217626 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.194732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40399 (local address 127.0.0.1:47608)
0809 06:19:51.208374 (+ 13642us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.208394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.209378 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.210043 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.210047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.210072 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:51.210432 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.210444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.212324 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.212330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.214573 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.214587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.216774 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.216780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.216959 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.216987 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.217064 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.217192 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13411,"mutex_wait_us":1,"spinlock_wait_cycles":6016,"thread_start_us":12523,"threads_started":1}
I20260809 06:19:51.218264 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.193517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.3.137.67:33331)
0809 06:19:51.207719 (+ 14202us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.207725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.207749 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:19:51.207831 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.207838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.208137 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.208399 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:51.209356 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.210766 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.213840 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.215400 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.217164 (+ 1764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.217362 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.217430 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.217616 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14061,"thread_start_us":13929,"threads_started":1}
I20260809 06:19:51.218343 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.195118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.3.137.66:40399)
0809 06:19:51.205690 (+ 10572us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.205698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.208647 (+ 2949us) server_negotiation.cc:407] Connection header received
0809 06:19:51.209607 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.209614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.209837 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.210029 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:51.210636 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.212164 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.214777 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.216296 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.216430 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.216619 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.217871 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.218042 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10413,"thread_start_us":10274,"threads_started":1}
I20260809 06:19:51.218576 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.194346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:39019 (local address 127.0.0.1:55244)
0809 06:19:51.207785 (+ 13439us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.207807 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.207955 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.208763 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.208768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.208846 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:51.209296 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.209305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.211287 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.211297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.213530 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.213548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.217604 (+ 4056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.217617 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.217829 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.217877 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.217974 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.218120 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13243,"mutex_wait_us":3,"spinlock_wait_cycles":8448,"thread_start_us":13123,"threads_started":1}
I20260809 06:19:51.219501 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.194640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.3.137.65:39019)
0809 06:19:51.207128 (+ 12488us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.207141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.207922 (+ 781us) server_negotiation.cc:407] Connection header received
0809 06:19:51.208200 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.208209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.208570 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.208769 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:51.209472 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.211122 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.213737 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.216115 (+ 2378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.216271 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.216522 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.218930 (+ 2408us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.219139 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12249,"thread_start_us":12071,"threads_started":1}
I20260809 06:19:51.222368 12280 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 (DEFAULT_TABLE table=test-table [id=0cfcb1c712e843f59e0656661657c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.223958 12280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6d70333fdc4736a97dbdaa4bfea982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.224077 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 (DEFAULT_TABLE table=test-table [id=0cfcb1c712e843f59e0656661657c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.225391 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6d70333fdc4736a97dbdaa4bfea982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.224882 11975 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 (DEFAULT_TABLE table=test-table [id=0cfcb1c712e843f59e0656661657c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.226663 11975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6d70333fdc4736a97dbdaa4bfea982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.238219 12382 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Bootstrap starting.
I20260809 06:19:51.241477 12384 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Bootstrap starting.
I20260809 06:19:51.242571 12382 tablet_bootstrap.cc:654] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.243402 12383 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Bootstrap starting.
I20260809 06:19:51.243956 12382 log.cc:824] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:51.245213 12384 tablet_bootstrap.cc:654] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.246618 12383 tablet_bootstrap.cc:654] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.246918 12384 log.cc:824] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:51.247872 12383 log.cc:824] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:51.255067 12382 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: No bootstrap required, opened a new log
I20260809 06:19:51.255419 12384 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: No bootstrap required, opened a new log
I20260809 06:19:51.255422 12382 ts_tablet_manager.cc:1397] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20260809 06:19:51.255784 12384 ts_tablet_manager.cc:1397] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260809 06:19:51.266362 12382 raft_consensus.cc:348] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.266984 12382 raft_consensus.cc:374] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.267160 12382 raft_consensus.cc:729] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.267976 12382 consensus_queue.cc:260] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.268203 12384 raft_consensus.cc:348] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.268667 12384 raft_consensus.cc:374] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.268844 12384 raft_consensus.cc:729] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.269728 12384 consensus_queue.cc:260] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.273813 12383 tablet_bootstrap.cc:492] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: No bootstrap required, opened a new log
I20260809 06:19:51.274132 12383 ts_tablet_manager.cc:1397] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.022s
I20260809 06:19:51.277055 12382 ts_tablet_manager.cc:1428] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260809 06:19:51.278779 12384 ts_tablet_manager.cc:1428] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260809 06:19:51.283938 12383 raft_consensus.cc:348] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.284262 12383 raft_consensus.cc:374] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.284356 12383 raft_consensus.cc:729] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa3eae3ccc4105b3551349d7e4e336, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.284798 12383 consensus_queue.cc:260] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.286316 12383 ts_tablet_manager.cc:1428] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260809 06:19:51.366044 12388 raft_consensus.cc:482] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:51.366393 12388 raft_consensus.cc:504] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.368160 12388 leader_election.cc:288] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331), 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399)
I20260809 06:19:51.375293 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.368496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:42769 (local address 127.3.137.67:33331)
0809 06:19:51.368769 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.368779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.369784 (+ 1005us) server_negotiation.cc:407] Connection header received
0809 06:19:51.369989 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.370001 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.370071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.370169 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:51.370841 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.371623 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.373459 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.374270 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.374453 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.374543 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.374881 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.375039 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:19:51.375288 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.368736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:37323 (local address 127.3.137.66:40399)
0809 06:19:51.368978 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.368987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.369688 (+ 701us) server_negotiation.cc:407] Connection header received
0809 06:19:51.369871 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.369876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.369954 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.370053 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:51.370696 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.371502 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.373484 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.374243 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.374459 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.374538 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.374878 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.375036 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:19:51.375471 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.368436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40399 (local address 127.3.137.65:37323)
0809 06:19:51.369571 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.369586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.369709 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.370072 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.370076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.370100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:51.370538 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.370547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.371631 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.371639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.373319 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.373331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.374501 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.374507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.374658 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.374697 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.374845 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.375017 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":93,"thread_start_us":809,"threads_started":1}
I20260809 06:19:51.375617 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.368519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33331 (local address 127.3.137.65:42769)
0809 06:19:51.369668 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.369692 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.369834 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.370194 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.370199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.370237 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:51.370671 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.370679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.371774 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.371780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.373294 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.373309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.374546 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.374553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.374722 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.374745 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.374854 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.375034 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":68,"thread_start_us":479,"threads_started":1}
I20260809 06:19:51.376571 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6d70333fdc4736a97dbdaa4bfea982" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41" is_pre_election: true
I20260809 06:19:51.376695 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6d70333fdc4736a97dbdaa4bfea982" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336" is_pre_election: true
I20260809 06:19:51.377177 12307 raft_consensus.cc:2454] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.377182 12153 raft_consensus.cc:2454] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.377800 11923 leader_election.cc:302] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.378208 12388 raft_consensus.cc:2781] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:51.378377 12388 raft_consensus.cc:482] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:51.378494 12388 raft_consensus.cc:3037] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.380753 12388 raft_consensus.cc:504] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.381402 12388 leader_election.cc:288] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Requested vote from peers daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331), 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399)
I20260809 06:19:51.381749 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6d70333fdc4736a97dbdaa4bfea982" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336"
I20260809 06:19:51.381771 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6d70333fdc4736a97dbdaa4bfea982" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41"
I20260809 06:19:51.381968 12153 raft_consensus.cc:3037] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.381963 12307 raft_consensus.cc:3037] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.385245 12307 raft_consensus.cc:2454] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.385257 12153 raft_consensus.cc:2454] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.385679 11922 leader_election.cc:302] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: daaa3eae3ccc4105b3551349d7e4e336, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.385986 12388 raft_consensus.cc:2781] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:51.386431 12388 raft_consensus.cc:686] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 LEADER]: Becoming Leader. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Running, Role: LEADER
I20260809 06:19:51.386919 12388 consensus_queue.cc:237] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } }
I20260809 06:19:51.391757 11825 catalog_manager.cc:5495] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } health_report { overall_health: UNKNOWN } } }
W20260809 06:19:51.398993 12055 tablet.cc:2369] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:51.412132 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.406614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.3.137.65:39019)
0809 06:19:51.406819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.406825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.406932 (+ 107us) server_negotiation.cc:407] Connection header received
0809 06:19:51.407241 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.407246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.407344 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.407512 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:51.407880 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.408583 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.409942 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.410367 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.410518 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.410946 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:51.411539 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:51.411696 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.411780 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.411862 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:19:51.420193 12153 raft_consensus.cc:1261] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Refusing update from remote peer fd3b1eb34c20483993c4cc5dedc58de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:51.420203 12307 raft_consensus.cc:1261] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Refusing update from remote peer fd3b1eb34c20483993c4cc5dedc58de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:51.421276 12388 consensus_queue.cc:1035] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:51.421581 12393 consensus_queue.cc:1035] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:51.438376 12397 mvcc.cc:205] Tried to move back new op lower bound from 7316506179247038464 to 7316506179124772864. Current Snapshot: MvccSnapshot[applied={T|T < 7316506179247038464}]
I20260809 06:19:51.448462 11823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:51.448711 11823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:51.452481 11823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Sending DeleteTablet for 3 replicas of tablet 4c6d70333fdc4736a97dbdaa4bfea982
I20260809 06:19:51.453828 12280 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:51 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:56836
I20260809 06:19:51.453828 11975 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:51 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55244
I20260809 06:19:51.453896 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c6d70333fdc4736a97dbdaa4bfea982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:51 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47608
I20260809 06:19:51.457718 11825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:51.458386 11825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:51.467052 12127 tablet_service.cc:1467] Processing CreateTablet for tablet 55055d6b1dc041e2904ea49b63504bed (DEFAULT_TABLE table=test-table [id=b0bb68775515454582211e16daf5db80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.467702 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55055d6b1dc041e2904ea49b63504bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.472504 11974 tablet_service.cc:1467] Processing CreateTablet for tablet 55055d6b1dc041e2904ea49b63504bed (DEFAULT_TABLE table=test-table [id=b0bb68775515454582211e16daf5db80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.473138 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55055d6b1dc041e2904ea49b63504bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.475768 12282 tablet_service.cc:1467] Processing CreateTablet for tablet 55055d6b1dc041e2904ea49b63504bed (DEFAULT_TABLE table=test-table [id=b0bb68775515454582211e16daf5db80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:51.476423 12282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55055d6b1dc041e2904ea49b63504bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.483371 12383 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Bootstrap starting.
I20260809 06:19:51.484998 12383 tablet_bootstrap.cc:654] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.485316 12384 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Bootstrap starting.
I20260809 06:19:51.487856 12384 tablet_bootstrap.cc:654] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.493522 12384 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: No bootstrap required, opened a new log
I20260809 06:19:51.493762 12384 ts_tablet_manager.cc:1397] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:19:51.494550 12382 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Bootstrap starting.
I20260809 06:19:51.496331 12382 tablet_bootstrap.cc:654] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.497849 12384 raft_consensus.cc:348] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.498405 12384 raft_consensus.cc:374] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.498812 12384 raft_consensus.cc:729] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.499788 12384 consensus_queue.cc:260] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.501030 12384 ts_tablet_manager.cc:1428] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260809 06:19:51.501938 12406 tablet_replica.cc:330] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: stopping tablet replica
I20260809 06:19:51.502681 12409 tablet_replica.cc:330] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: stopping tablet replica
I20260809 06:19:51.502828 12383 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: No bootstrap required, opened a new log
I20260809 06:19:51.503436 12383 ts_tablet_manager.cc:1397] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20260809 06:19:51.504277 12383 raft_consensus.cc:348] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.504515 12383 raft_consensus.cc:374] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.504603 12383 raft_consensus.cc:729] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa3eae3ccc4105b3551349d7e4e336, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.504889 12383 consensus_queue.cc:260] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.506796 12383 ts_tablet_manager.cc:1428] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 06:19:51.507356 12406 raft_consensus.cc:2229] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:51.507711 12409 raft_consensus.cc:2229] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:51.508358 12409 raft_consensus.cc:2258] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:51.508885 12406 raft_consensus.cc:2258] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:51.510018 12408 tablet_replica.cc:330] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: stopping tablet replica
I20260809 06:19:51.510527 12408 raft_consensus.cc:2229] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:51.511656 12406 ts_tablet_manager.cc:1905] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:51.511791 12408 raft_consensus.cc:2258] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:51.513425 12409 ts_tablet_manager.cc:1905] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:51.516185 12382 tablet_bootstrap.cc:492] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: No bootstrap required, opened a new log
I20260809 06:19:51.516433 12382 ts_tablet_manager.cc:1397] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
W20260809 06:19:51.517060 12358 tablet.cc:2369] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:51.517660 12382 raft_consensus.cc:348] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.518026 12382 raft_consensus.cc:374] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.518208 12382 raft_consensus.cc:729] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.518638 12382 consensus_queue.cc:260] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.520718 12409 ts_tablet_manager.cc:1918] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:51.520929 12409 log.cc:1196] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/4c6d70333fdc4736a97dbdaa4bfea982
I20260809 06:19:51.521564 12409 ts_tablet_manager.cc:1939] T 4c6d70333fdc4736a97dbdaa4bfea982 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting consensus metadata
I20260809 06:19:51.522027 12406 ts_tablet_manager.cc:1918] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:51.522260 12406 log.cc:1196] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/4c6d70333fdc4736a97dbdaa4bfea982
I20260809 06:19:51.522905 12406 ts_tablet_manager.cc:1939] T 4c6d70333fdc4736a97dbdaa4bfea982 P daaa3eae3ccc4105b3551349d7e4e336: Deleting consensus metadata
I20260809 06:19:51.523205 12408 ts_tablet_manager.cc:1905] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:51.523586 11811 catalog_manager.cc:4841] TS 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399): tablet 4c6d70333fdc4736a97dbdaa4bfea982 (table test-table [id=0cfcb1c712e843f59e0656661657c9fa]) successfully deleted
I20260809 06:19:51.523916 12382 ts_tablet_manager.cc:1428] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260809 06:19:51.524744 11810 catalog_manager.cc:4841] TS daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331): tablet 4c6d70333fdc4736a97dbdaa4bfea982 (table test-table [id=0cfcb1c712e843f59e0656661657c9fa]) successfully deleted
I20260809 06:19:51.531957 12408 ts_tablet_manager.cc:1918] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:51.532135 12408 log.cc:1196] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/4c6d70333fdc4736a97dbdaa4bfea982
I20260809 06:19:51.532727 12408 ts_tablet_manager.cc:1939] T 4c6d70333fdc4736a97dbdaa4bfea982 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting consensus metadata
I20260809 06:19:51.534261 11810 catalog_manager.cc:4841] TS fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019): tablet 4c6d70333fdc4736a97dbdaa4bfea982 (table test-table [id=0cfcb1c712e843f59e0656661657c9fa]) successfully deleted
I20260809 06:19:51.600345 12388 raft_consensus.cc:482] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:51.600576 12388 raft_consensus.cc:504] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.601318 12388 leader_election.cc:288] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:51.601965 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55055d6b1dc041e2904ea49b63504bed" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41" is_pre_election: true
I20260809 06:19:51.601960 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55055d6b1dc041e2904ea49b63504bed" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336" is_pre_election: true
I20260809 06:19:51.602244 12153 raft_consensus.cc:2454] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.602257 12307 raft_consensus.cc:2454] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.602699 11923 leader_election.cc:302] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.603058 12388 raft_consensus.cc:2781] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:51.603222 12388 raft_consensus.cc:482] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:51.603372 12388 raft_consensus.cc:3037] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.605602 12388 raft_consensus.cc:504] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.606209 12388 leader_election.cc:288] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Requested vote from peers 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:51.606653 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55055d6b1dc041e2904ea49b63504bed" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41"
I20260809 06:19:51.606673 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55055d6b1dc041e2904ea49b63504bed" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336"
I20260809 06:19:51.606868 12307 raft_consensus.cc:3037] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.606868 12153 raft_consensus.cc:3037] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.609822 12153 raft_consensus.cc:2454] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.609822 12307 raft_consensus.cc:2454] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.610189 11922 leader_election.cc:302] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: daaa3eae3ccc4105b3551349d7e4e336, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.610492 12388 raft_consensus.cc:2781] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:51.610659 12388 raft_consensus.cc:686] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 LEADER]: Becoming Leader. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Running, Role: LEADER
I20260809 06:19:51.611014 12388 consensus_queue.cc:237] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.620631 11823 catalog_manager.cc:5495] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:19:51 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256388, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.126@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:51.688213 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.673400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.3.137.126:39717)
0809 06:19:51.674333 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:51.674342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:51.674367 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:19:51.674419 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:51.674422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:51.674473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:51.674584 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:51.676196 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.677074 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.678355 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.678745 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.678881 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.682617 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:51.682622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:51.682625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:51.682654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:51.685389 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:51.686086 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:51.686092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:51.686095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:51.686218 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:51.686595 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:51.686601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:51.686604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:51.686864 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:51.687071 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:51.687470 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 06:19:51.687838 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":805,"thread_start_us":583,"threads_started":1}
I20260809 06:19:51.688236 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:51.673254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39717 (local address 127.0.0.1:41826)
0809 06:19:51.674064 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:51.674090 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:51.674265 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:51.674618 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:51.674625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:51.675616 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:51.675997 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.676008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.677197 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.677205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.678192 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:51.678204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:51.678935 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:51.678940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:51.679154 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:51.679850 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:51.679872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:51.682416 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:51.685520 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:51.685535 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:51.685543 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:51.685960 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:51.686326 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:51.686333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:51.686338 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:51.686486 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:51.687026 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:51.687035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:51.687219 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 06:19:51.687386 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 06:19:51.687679 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":362,"threads_started":1}
I20260809 06:19:51.693853 11823 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:41826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:19:51.694867 11823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:51.702291 11975 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c178f140449308cb5f4f9ec66ffb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04dfe985c15841ad87bac1ca49a9fa91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:51.702638 12280 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c178f140449308cb5f4f9ec66ffb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04dfe985c15841ad87bac1ca49a9fa91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:51.702943 11975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c178f140449308cb5f4f9ec66ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.703300 12280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c178f140449308cb5f4f9ec66ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.708833 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4c178f140449308cb5f4f9ec66ffb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04dfe985c15841ad87bac1ca49a9fa91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:19:51.709489 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4c178f140449308cb5f4f9ec66ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:51.709573 12382 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: Bootstrap starting.
I20260809 06:19:51.711704 12382 tablet_bootstrap.cc:654] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: Neither blocks nor log segments found. Creating new log.
W20260809 06:19:51.712137 12208 tablet.cc:2369] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:19:51.716171 12383 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: Bootstrap starting.
I20260809 06:19:51.717466 12382 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: No bootstrap required, opened a new log
I20260809 06:19:51.717713 12382 ts_tablet_manager.cc:1397] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260809 06:19:51.717949 12384 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: Bootstrap starting.
I20260809 06:19:51.718166 12383 tablet_bootstrap.cc:654] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.718752 12382 raft_consensus.cc:348] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.719182 12382 raft_consensus.cc:374] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.719323 12382 raft_consensus.cc:729] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.720132 12384 tablet_bootstrap.cc:654] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:51.720992 12382 consensus_queue.cc:260] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.721577 12388 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:51.722019 12388 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:51.722679 12382 ts_tablet_manager.cc:1428] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 06:19:51.729571 12384 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: No bootstrap required, opened a new log
I20260809 06:19:51.729799 12384 ts_tablet_manager.cc:1397] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260809 06:19:51.729996 12383 tablet_bootstrap.cc:492] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: No bootstrap required, opened a new log
I20260809 06:19:51.730269 12383 ts_tablet_manager.cc:1397] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20260809 06:19:51.730633 12384 raft_consensus.cc:348] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.730903 12384 raft_consensus.cc:374] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.731020 12384 raft_consensus.cc:729] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.731052 12383 raft_consensus.cc:348] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.731371 12383 raft_consensus.cc:374] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:51.731479 12383 raft_consensus.cc:729] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa3eae3ccc4105b3551349d7e4e336, State: Initialized, Role: FOLLOWER
I20260809 06:19:51.731307 12384 consensus_queue.cc:260] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.731716 12383 consensus_queue.cc:260] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.731859 12389 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:51.732266 12389 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:51.732388 12383 ts_tablet_manager.cc:1428] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:19:51.732628 12390 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:51.733023 12390 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:51.734453 12384 ts_tablet_manager.cc:1428] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260809 06:19:51.941645 12388 raft_consensus.cc:482] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:51.941879 12388 raft_consensus.cc:504] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.942507 12388 leader_election.cc:288] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:51.943173 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c178f140449308cb5f4f9ec66ffb8" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41" is_pre_election: true
I20260809 06:19:51.943174 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c178f140449308cb5f4f9ec66ffb8" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336" is_pre_election: true
I20260809 06:19:51.943460 12153 raft_consensus.cc:2454] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.943460 12307 raft_consensus.cc:2454] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 0.
I20260809 06:19:51.943918 11922 leader_election.cc:302] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, daaa3eae3ccc4105b3551349d7e4e336, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.944242 12388 raft_consensus.cc:2781] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:51.944336 12388 raft_consensus.cc:482] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:51.944463 12388 raft_consensus.cc:3037] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.946502 12388 raft_consensus.cc:504] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.947187 12388 leader_election.cc:288] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Requested vote from peers 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:51.947641 12307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c178f140449308cb5f4f9ec66ffb8" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336"
I20260809 06:19:51.947641 12153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4c178f140449308cb5f4f9ec66ffb8" candidate_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caa226cb74d41daa93c098cb1eaeb41"
I20260809 06:19:51.947889 12307 raft_consensus.cc:3037] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.947909 12153 raft_consensus.cc:3037] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:51.950824 12153 raft_consensus.cc:2454] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.950814 12307 raft_consensus.cc:2454] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b1eb34c20483993c4cc5dedc58de9 in term 1.
I20260809 06:19:51.951337 11923 leader_election.cc:302] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:51.951643 12388 raft_consensus.cc:2781] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:51.951823 12388 raft_consensus.cc:686] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 LEADER]: Becoming Leader. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Running, Role: LEADER
I20260809 06:19:51.952093 12388 consensus_queue.cc:237] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:51.953719 12401 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: TxnStatusTablet state changed. Reason: New leader fd3b1eb34c20483993c4cc5dedc58de9. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:51.953976 12401 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:51.954619 12427 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:19:51.954895 12427 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:19:51.955402 11823 catalog_manager.cc:5495] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:52.010044 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.003910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:39019 (local address 127.0.0.1:55262)
0809 06:19:52.004203 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.004232 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.004389 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.004908 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.004911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.004921 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:52.005092 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.005100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.006768 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.006772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.007834 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.007847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.008867 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.008871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.009026 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.009086 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.009596 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.009618 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.009687 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.009808 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:19:52.010335 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.003997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.3.137.65:39019)
0809 06:19:52.004569 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.004575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.004596 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:52.004675 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.004681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.004765 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.004867 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:52.005251 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.006632 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.007976 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.008693 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.008826 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.009283 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.009483 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.009580 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.009737 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.009827 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":273,"threads_started":1}
I20260809 06:19:52.014782 12307 raft_consensus.cc:1261] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Refusing update from remote peer fd3b1eb34c20483993c4cc5dedc58de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.015012 12153 raft_consensus.cc:1261] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Refusing update from remote peer fd3b1eb34c20483993c4cc5dedc58de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.015740 12401 consensus_queue.cc:1035] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.016311 12388 consensus_queue.cc:1035] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.019129 12388 consensus_queue.cc:1035] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.036872 12390 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: TxnStatusTablet state changed. Reason: New leader fd3b1eb34c20483993c4cc5dedc58de9. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.037175 12390 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:52.041126 12430 mvcc.cc:205] Tried to move back new op lower bound from 7316506181685903360 to 7316506181438242816. Current Snapshot: MvccSnapshot[applied={T|T < 7316506181685903360}]
I20260809 06:19:52.044062 12432 mvcc.cc:205] Tried to move back new op lower bound from 7316506181685903360 to 7316506181438242816. Current Snapshot: MvccSnapshot[applied={T|T < 7316506181685903360}]
I20260809 06:19:52.047887 12388 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.048233 12388 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:52.052249 12389 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: TxnStatusTablet state changed. Reason: New leader fd3b1eb34c20483993c4cc5dedc58de9. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.052589 12389 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:52.060331 12390 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.060595 12390 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P daaa3eae3ccc4105b3551349d7e4e336: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:52.076321 12388 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.076753 12388 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P fd3b1eb34c20483993c4cc5dedc58de9: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:19:52.081948 12401 consensus_queue.cc:1035] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.099143 12433 tablet_replica.cc:439] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } } }
I20260809 06:19:52.099507 12433 tablet_replica.cc:442] T 8f4c178f140449308cb5f4f9ec66ffb8 P 4caa226cb74d41daa93c098cb1eaeb41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:19:52.105978 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.095079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:50467 (local address 127.3.137.65:39019)
0809 06:19:52.095281 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.095288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.099562 (+ 4274us) server_negotiation.cc:407] Connection header received
0809 06:19:52.099854 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.099860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.099950 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.100018 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:52.100989 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.102070 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.104054 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.104520 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.104663 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.105164 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.105281 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.105396 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.105516 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.105597 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:19:52.106101 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.093780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:39019 (local address 127.3.137.65:50467)
0809 06:19:52.099451 (+ 5671us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.099478 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.099690 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.100264 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.100268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.100282 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:52.100768 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.100781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.102242 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.102248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.103893 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.103907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.104715 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.104723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.104892 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.104953 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.105394 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.105406 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.105498 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.105643 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5486,"thread_start_us":5343,"threads_started":1}
I20260809 06:19:52.172433 11825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.172693 11825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.175673 11825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Sending DeleteTablet for 3 replicas of tablet 55055d6b1dc041e2904ea49b63504bed
I20260809 06:19:52.176476 11974 tablet_service.cc:1514] Processing DeleteTablet for tablet 55055d6b1dc041e2904ea49b63504bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55244
I20260809 06:19:52.176913 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet 55055d6b1dc041e2904ea49b63504bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47608
I20260809 06:19:52.177345 12451 tablet_replica.cc:330] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: stopping tablet replica
I20260809 06:19:52.177404 12280 tablet_service.cc:1514] Processing DeleteTablet for tablet 55055d6b1dc041e2904ea49b63504bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:56836
I20260809 06:19:52.177672 12452 tablet_replica.cc:330] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: stopping tablet replica
I20260809 06:19:52.177868 12452 raft_consensus.cc:2229] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.178120 12452 raft_consensus.cc:2258] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.178175 12451 raft_consensus.cc:2229] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:52.178745 12451 raft_consensus.cc:2258] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.179334 12452 ts_tablet_manager.cc:1905] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.180014 12451 ts_tablet_manager.cc:1905] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.186578 12452 ts_tablet_manager.cc:1918] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:52.186723 12452 log.cc:1196] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/55055d6b1dc041e2904ea49b63504bed
I20260809 06:19:52.187261 12453 tablet_replica.cc:330] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: stopping tablet replica
I20260809 06:19:52.187260 12452 ts_tablet_manager.cc:1939] T 55055d6b1dc041e2904ea49b63504bed P 4caa226cb74d41daa93c098cb1eaeb41: Deleting consensus metadata
I20260809 06:19:52.187572 12453 raft_consensus.cc:2229] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.187901 12453 raft_consensus.cc:2258] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.188464 11811 catalog_manager.cc:4841] TS 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399): tablet 55055d6b1dc041e2904ea49b63504bed (table test-table [id=b0bb68775515454582211e16daf5db80]) successfully deleted
I20260809 06:19:52.189227 12453 ts_tablet_manager.cc:1905] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.189407 12451 ts_tablet_manager.cc:1918] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:52.189581 12451 log.cc:1196] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/55055d6b1dc041e2904ea49b63504bed
I20260809 06:19:52.190060 12451 ts_tablet_manager.cc:1939] T 55055d6b1dc041e2904ea49b63504bed P fd3b1eb34c20483993c4cc5dedc58de9: Deleting consensus metadata
I20260809 06:19:52.191176 11810 catalog_manager.cc:4841] TS fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019): tablet 55055d6b1dc041e2904ea49b63504bed (table test-table [id=b0bb68775515454582211e16daf5db80]) successfully deleted
I20260809 06:19:52.196838 12453 ts_tablet_manager.cc:1918] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:52.196978 12453 log.cc:1196] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/55055d6b1dc041e2904ea49b63504bed
I20260809 06:19:52.197503 12453 ts_tablet_manager.cc:1939] T 55055d6b1dc041e2904ea49b63504bed P daaa3eae3ccc4105b3551349d7e4e336: Deleting consensus metadata
I20260809 06:19:52.198688 11810 catalog_manager.cc:4841] TS daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331): tablet 55055d6b1dc041e2904ea49b63504bed (table test-table [id=b0bb68775515454582211e16daf5db80]) successfully deleted
Aug 09 06:19:52 dist-test-slave-2w3w krb5kdc[11777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:19:52 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:19:52 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:19:52.279348 12461 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:19:52.249104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.3.137.126:39717)
0809 06:19:52.262884 (+ 13780us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.262889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.263254 (+ 365us) server_negotiation.cc:407] Connection header received
0809 06:19:52.263521 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.263528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.263612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.263732 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:52.265187 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.265964 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.267358 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.267790 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.267974 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.277171 (+ 9197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41828: BlockingRecv error: recv got EOF from 127.0.0.1:41828 (error 108)
Metrics: {"server-negotiator.queue_time_us":13646,"thread_start_us":13494,"threads_started":1}
Aug 09 06:19:52 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM
I20260809 06:19:52.296123 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.277336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.3.137.126:39717)
0809 06:19:52.279157 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.279164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.279666 (+ 502us) server_negotiation.cc:407] Connection header received
0809 06:19:52.279852 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.279856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.279913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.280002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:52.281686 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.282545 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.284360 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.284744 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.284840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.289634 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:52.289641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:52.289646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:52.289696 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:52.293430 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:52.294103 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:52.294108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:52.294110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:52.294253 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:52.294672 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:52.294677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:52.294682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:52.295060 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:52.295319 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.295630 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.295803 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1503,"threads_started":1}
I20260809 06:19:52.299965 11825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:19:52.300588 11825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:52.307731 11974 tablet_service.cc:1467] Processing CreateTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 (DEFAULT_TABLE table=test-table [id=89df6bcb28904815a2ef18f2bad216f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.308190 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cdc289d01142dfbb9b123ed8a8d202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.308218 12128 tablet_service.cc:1467] Processing CreateTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 (DEFAULT_TABLE table=test-table [id=89df6bcb28904815a2ef18f2bad216f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.308658 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cdc289d01142dfbb9b123ed8a8d202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.311751 12280 tablet_service.cc:1467] Processing CreateTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 (DEFAULT_TABLE table=test-table [id=89df6bcb28904815a2ef18f2bad216f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.312191 12280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cdc289d01142dfbb9b123ed8a8d202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.323354 12469 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Bootstrap starting.
I20260809 06:19:52.324120 12470 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Bootstrap starting.
I20260809 06:19:52.325151 12469 tablet_bootstrap.cc:654] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.326077 12470 tablet_bootstrap.cc:654] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.327586 12468 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Bootstrap starting.
I20260809 06:19:52.328370 12469 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: No bootstrap required, opened a new log
I20260809 06:19:52.328518 12469 ts_tablet_manager.cc:1397] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 06:19:52.329314 12469 raft_consensus.cc:348] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.329591 12469 raft_consensus.cc:374] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.329703 12469 raft_consensus.cc:729] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.329995 12469 consensus_queue.cc:260] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.330214 12468 tablet_bootstrap.cc:654] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.330569 12469 ts_tablet_manager.cc:1428] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 06:19:52.333202 12470 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: No bootstrap required, opened a new log
I20260809 06:19:52.333451 12470 ts_tablet_manager.cc:1397] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:19:52.334537 12470 raft_consensus.cc:348] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.335011 12470 raft_consensus.cc:374] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.335183 12470 raft_consensus.cc:729] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa3eae3ccc4105b3551349d7e4e336, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.335479 12470 consensus_queue.cc:260] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.336208 12470 ts_tablet_manager.cc:1428] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:52.339448 12468 tablet_bootstrap.cc:492] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: No bootstrap required, opened a new log
I20260809 06:19:52.339635 12468 ts_tablet_manager.cc:1397] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260809 06:19:52.340385 12468 raft_consensus.cc:348] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.340567 12468 raft_consensus.cc:374] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.340687 12468 raft_consensus.cc:729] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.340951 12468 consensus_queue.cc:260] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.341511 12468 ts_tablet_manager.cc:1428] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:19:52.395649 12433 raft_consensus.cc:482] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:52.395889 12433 raft_consensus.cc:504] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.397558 12433 leader_election.cc:288] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:52.408761 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.397880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:53907 (local address 127.3.137.65:39019)
0809 06:19:52.398162 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.398172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.398823 (+ 651us) server_negotiation.cc:407] Connection header received
0809 06:19:52.399168 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.399181 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.399304 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.399383 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:52.400487 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.402140 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.405722 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.407610 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.407785 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.407865 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.408277 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.408446 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 06:19:52.409569 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.397885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:39019 (local address 127.3.137.66:53907)
0809 06:19:52.398723 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.398745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.398939 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.399653 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.399661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.399695 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:52.400226 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.400243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.402427 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.402435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.405479 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.405501 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.407831 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.407847 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.408039 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.408080 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.409046 (+ 966us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.409193 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":487,"threads_started":1}
I20260809 06:19:52.410619 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cdc289d01142dfbb9b123ed8a8d202" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" is_pre_election: true
I20260809 06:19:52.411193 12003 raft_consensus.cc:2454] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 0.
I20260809 06:19:52.411854 12077 leader_election.cc:302] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:52.412322 12433 raft_consensus.cc:2781] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:52.412474 12433 raft_consensus.cc:482] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:52.412580 12433 raft_consensus.cc:3037] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.414747 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.398864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33331 (local address 127.3.137.66:39715)
0809 06:19:52.407542 (+ 8678us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.407576 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.407726 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.408113 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.408118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.408144 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:19:52.408531 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.408540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.409954 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.409962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.412044 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.412059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.413841 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.413848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.414039 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.414072 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.414192 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.414339 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8512,"thread_start_us":8317,"threads_started":1}
I20260809 06:19:52.414736 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.399049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39715 (local address 127.3.137.67:33331)
0809 06:19:52.407803 (+ 8754us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.407809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.407828 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:19:52.407900 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.407905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.407977 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.408096 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:19:52.408708 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.409587 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.412224 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.413406 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.413607 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.413677 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.414219 (+ 542us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.414356 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8603,"thread_start_us":8439,"threads_started":1}
I20260809 06:19:52.415634 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cdc289d01142dfbb9b123ed8a8d202" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336" is_pre_election: true
I20260809 06:19:52.415925 12301 raft_consensus.cc:2454] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 0.
I20260809 06:19:52.416090 12433 raft_consensus.cc:504] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.416853 12433 leader_election.cc:288] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 election: Requested vote from peers fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:52.417372 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cdc289d01142dfbb9b123ed8a8d202" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336"
I20260809 06:19:52.417342 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cdc289d01142dfbb9b123ed8a8d202" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b1eb34c20483993c4cc5dedc58de9"
I20260809 06:19:52.417568 12301 raft_consensus.cc:3037] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.417573 12003 raft_consensus.cc:3037] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.420039 12301 raft_consensus.cc:2454] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 1.
I20260809 06:19:52.420430 12077 leader_election.cc:302] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, daaa3eae3ccc4105b3551349d7e4e336; no voters:
I20260809 06:19:52.421062 12003 raft_consensus.cc:2454] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 1.
I20260809 06:19:52.421407 12433 raft_consensus.cc:2781] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:52.421584 12433 raft_consensus.cc:686] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 LEADER]: Becoming Leader. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Running, Role: LEADER
I20260809 06:19:52.422003 12433 consensus_queue.cc:237] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.425961 11823 catalog_manager.cc:5495] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "4caa226cb74d41daa93c098cb1eaeb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:52.449757 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.443814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.3.137.66:40399)
0809 06:19:52.444422 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.444426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.444444 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:19:52.444524 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.444528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.444597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.444767 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:52.445188 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.445969 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.447401 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.447849 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.447990 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.448482 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.449072 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.449234 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.449356 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.449433 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":359,"threads_started":1}
I20260809 06:19:52.453545 12301 raft_consensus.cc:1261] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Refusing update from remote peer 4caa226cb74d41daa93c098cb1eaeb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.453945 12003 raft_consensus.cc:1261] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Refusing update from remote peer 4caa226cb74d41daa93c098cb1eaeb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.454317 12389 consensus_queue.cc:1035] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.454777 12433 consensus_queue.cc:1035] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.481241 11823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.481534 11823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.485875 11823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Sending DeleteTablet for 3 replicas of tablet f4cdc289d01142dfbb9b123ed8a8d202
I20260809 06:19:52.486624 11974 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55244
I20260809 06:19:52.486881 12451 tablet_replica.cc:330] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: stopping tablet replica
I20260809 06:19:52.487090 12451 raft_consensus.cc:2229] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.487323 12451 raft_consensus.cc:2258] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.488503 12280 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:56836
I20260809 06:19:52.489064 12451 ts_tablet_manager.cc:1905] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.491117 12453 tablet_replica.cc:330] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: stopping tablet replica
I20260809 06:19:52.491338 12453 raft_consensus.cc:2229] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.491570 12453 raft_consensus.cc:2258] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.492462 11822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 06:19:52.493041 12453 ts_tablet_manager.cc:1905] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 06:19:52.493212 11822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:19:52.495814 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cdc289d01142dfbb9b123ed8a8d202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47608
I20260809 06:19:52.496138 12452 tablet_replica.cc:330] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: stopping tablet replica
I20260809 06:19:52.496337 12452 raft_consensus.cc:2229] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:52.496881 12452 raft_consensus.cc:2258] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.499182 12452 ts_tablet_manager.cc:1905] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.504117 11974 tablet_service.cc:1467] Processing CreateTablet for tablet e586a32d183f4ed98486b1ee5758a337 (DEFAULT_TABLE table=test-table [id=8bfb7f728e1247f2827002706cd04d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.504565 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e586a32d183f4ed98486b1ee5758a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.504791 12280 tablet_service.cc:1467] Processing CreateTablet for tablet e586a32d183f4ed98486b1ee5758a337 (DEFAULT_TABLE table=test-table [id=8bfb7f728e1247f2827002706cd04d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.505255 12280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e586a32d183f4ed98486b1ee5758a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.505193 12128 tablet_service.cc:1467] Processing CreateTablet for tablet e586a32d183f4ed98486b1ee5758a337 (DEFAULT_TABLE table=test-table [id=8bfb7f728e1247f2827002706cd04d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:19:52.505606 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e586a32d183f4ed98486b1ee5758a337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:52.510979 12468 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Bootstrap starting.
I20260809 06:19:52.512497 12468 tablet_bootstrap.cc:654] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.514801 12470 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Bootstrap starting.
I20260809 06:19:52.516314 12452 ts_tablet_manager.cc:1918] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:52.516489 12452 log.cc:1196] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/f4cdc289d01142dfbb9b123ed8a8d202
I20260809 06:19:52.516669 12470 tablet_bootstrap.cc:654] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.517015 12452 ts_tablet_manager.cc:1939] T f4cdc289d01142dfbb9b123ed8a8d202 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting consensus metadata
I20260809 06:19:52.518333 11811 catalog_manager.cc:4841] TS 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399): tablet f4cdc289d01142dfbb9b123ed8a8d202 (table test-table [id=89df6bcb28904815a2ef18f2bad216f3]) successfully deleted
I20260809 06:19:52.523584 12469 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Bootstrap starting.
I20260809 06:19:52.524911 12468 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: No bootstrap required, opened a new log
I20260809 06:19:52.525110 12468 ts_tablet_manager.cc:1397] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260809 06:19:52.525449 12469 tablet_bootstrap.cc:654] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Neither blocks nor log segments found. Creating new log.
I20260809 06:19:52.526077 12468 raft_consensus.cc:348] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.526401 12468 raft_consensus.cc:374] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.526500 12468 raft_consensus.cc:729] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b1eb34c20483993c4cc5dedc58de9, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.526865 12468 consensus_queue.cc:260] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.527652 12468 ts_tablet_manager.cc:1428] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 06:19:52.528101 12453 ts_tablet_manager.cc:1918] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:52.528296 12453 log.cc:1196] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/f4cdc289d01142dfbb9b123ed8a8d202
I20260809 06:19:52.528785 12453 ts_tablet_manager.cc:1939] T f4cdc289d01142dfbb9b123ed8a8d202 P daaa3eae3ccc4105b3551349d7e4e336: Deleting consensus metadata
I20260809 06:19:52.529985 11810 catalog_manager.cc:4841] TS daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331): tablet f4cdc289d01142dfbb9b123ed8a8d202 (table test-table [id=89df6bcb28904815a2ef18f2bad216f3]) successfully deleted
I20260809 06:19:52.530171 12451 ts_tablet_manager.cc:1918] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:19:52.530342 12451 log.cc:1196] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/f4cdc289d01142dfbb9b123ed8a8d202
I20260809 06:19:52.530809 12451 ts_tablet_manager.cc:1939] T f4cdc289d01142dfbb9b123ed8a8d202 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting consensus metadata
I20260809 06:19:52.531859 11810 catalog_manager.cc:4841] TS fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019): tablet f4cdc289d01142dfbb9b123ed8a8d202 (table test-table [id=89df6bcb28904815a2ef18f2bad216f3]) successfully deleted
I20260809 06:19:52.532927 12469 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: No bootstrap required, opened a new log
I20260809 06:19:52.533118 12469 ts_tablet_manager.cc:1397] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20260809 06:19:52.533344 12470 tablet_bootstrap.cc:492] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: No bootstrap required, opened a new log
I20260809 06:19:52.533468 12470 ts_tablet_manager.cc:1397] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.003s
I20260809 06:19:52.534003 12470 raft_consensus.cc:348] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.533939 12469 raft_consensus.cc:348] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.534186 12470 raft_consensus.cc:374] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.534247 12470 raft_consensus.cc:729] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa3eae3ccc4105b3551349d7e4e336, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.534226 12469 raft_consensus.cc:374] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:52.534318 12469 raft_consensus.cc:729] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Initialized, Role: FOLLOWER
I20260809 06:19:52.534497 12470 consensus_queue.cc:260] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.534564 12469 consensus_queue.cc:260] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.535069 12470 ts_tablet_manager.cc:1428] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:19:52.535148 12469 ts_tablet_manager.cc:1428] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:19:52.847586 12433 raft_consensus.cc:482] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:19:52.847853 12433 raft_consensus.cc:504] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.848765 12433 leader_election.cc:288] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:52.849344 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e586a32d183f4ed98486b1ee5758a337" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" is_pre_election: true
I20260809 06:19:52.849637 12003 raft_consensus.cc:2454] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 0.
I20260809 06:19:52.849795 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e586a32d183f4ed98486b1ee5758a337" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336" is_pre_election: true
I20260809 06:19:52.850086 12301 raft_consensus.cc:2454] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 0.
I20260809 06:19:52.850178 12077 leader_election.cc:302] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, fd3b1eb34c20483993c4cc5dedc58de9; no voters:
I20260809 06:19:52.850453 12433 raft_consensus.cc:2781] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:19:52.850622 12433 raft_consensus.cc:482] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:19:52.850731 12433 raft_consensus.cc:3037] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.854009 12433 raft_consensus.cc:504] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.854707 12433 leader_election.cc:288] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 election: Requested vote from peers fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019), daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331)
I20260809 06:19:52.855217 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e586a32d183f4ed98486b1ee5758a337" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b1eb34c20483993c4cc5dedc58de9"
I20260809 06:19:52.855325 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e586a32d183f4ed98486b1ee5758a337" candidate_uuid: "4caa226cb74d41daa93c098cb1eaeb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa3eae3ccc4105b3551349d7e4e336"
I20260809 06:19:52.855445 12003 raft_consensus.cc:3037] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.855563 12301 raft_consensus.cc:3037] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:52.857781 12301 raft_consensus.cc:2454] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 1.
I20260809 06:19:52.858209 12077 leader_election.cc:302] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4caa226cb74d41daa93c098cb1eaeb41, daaa3eae3ccc4105b3551349d7e4e336; no voters:
I20260809 06:19:52.858469 12003 raft_consensus.cc:2454] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4caa226cb74d41daa93c098cb1eaeb41 in term 1.
I20260809 06:19:52.858522 12433 raft_consensus.cc:2781] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:52.858686 12433 raft_consensus.cc:686] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 LEADER]: Becoming Leader. State: Replica: 4caa226cb74d41daa93c098cb1eaeb41, State: Running, Role: LEADER
I20260809 06:19:52.859004 12433 consensus_queue.cc:237] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } }
I20260809 06:19:52.862054 11822 catalog_manager.cc:5495] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "4caa226cb74d41daa93c098cb1eaeb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4caa226cb74d41daa93c098cb1eaeb41" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 } health_report { overall_health: UNKNOWN } } }
I20260809 06:19:52.925751 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.909976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:39019 (local address 127.3.137.66:60359)
0809 06:19:52.920789 (+ 10813us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.920812 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.920986 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.921365 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.921369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.921382 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:52.921577 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.921585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.922652 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.922657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.923750 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.923763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.924513 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.924519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.924662 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.924717 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.925139 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.925153 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.925255 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.925399 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10636,"thread_start_us":10550,"threads_started":1}
I20260809 06:19:52.925716 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.909931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:60359 (local address 127.3.137.65:39019)
0809 06:19:52.921037 (+ 11106us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.921045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.921066 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:19:52.921135 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.921148 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.921234 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.921344 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:52.921717 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.922508 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.923894 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.924312 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.924483 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.924917 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.925035 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.925136 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.925279 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.925357 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10991,"thread_start_us":11137,"threads_started":1}
I20260809 06:19:52.934289 12003 raft_consensus.cc:1261] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Refusing update from remote peer 4caa226cb74d41daa93c098cb1eaeb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.934558 12301 raft_consensus.cc:1261] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Refusing update from remote peer 4caa226cb74d41daa93c098cb1eaeb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:19:52.934894 12482 consensus_queue.cc:1035] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3b1eb34c20483993c4cc5dedc58de9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.935510 12433 consensus_queue.cc:1035] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daaa3eae3ccc4105b3551349d7e4e336" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:19:52.960708 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.955366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:44911 (local address 127.3.137.66:40399)
0809 06:19:52.955775 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.955782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.955799 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:19:52.955923 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.955929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.956003 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.956242 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:19:52.956685 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.957533 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.959257 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.959678 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.959786 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.960083 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.960212 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.960297 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:52.960387 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 06:19:52.960465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":254,"threads_started":1}
I20260809 06:19:52.960949 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.955088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40399 (local address 127.3.137.65:44911)
0809 06:19:52.955651 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:52.955673 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:52.955812 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:52.956246 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:52.956251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:52.956265 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:19:52.956537 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.956547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.957691 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.957697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.959118 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:52.959132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:52.959807 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:52.959811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:52.959918 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:52.959963 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:19:52.960310 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:19:52.960318 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:19:52.960374 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 06:19:52.960492 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":406,"threads_started":1}
I20260809 06:19:52.990906 11822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.991261 11822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:19:52.994104 11822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fff955601e204984a77be3cfc319f247: Sending DeleteTablet for 3 replicas of tablet e586a32d183f4ed98486b1ee5758a337
I20260809 06:19:52.994839 11974 tablet_service.cc:1514] Processing DeleteTablet for tablet e586a32d183f4ed98486b1ee5758a337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55244
I20260809 06:19:52.995015 12280 tablet_service.cc:1514] Processing DeleteTablet for tablet e586a32d183f4ed98486b1ee5758a337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:56836
I20260809 06:19:52.995188 12451 tablet_replica.cc:330] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: stopping tablet replica
I20260809 06:19:52.995291 12453 tablet_replica.cc:330] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: stopping tablet replica
I20260809 06:19:52.995385 12451 raft_consensus.cc:2229] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.995472 12453 raft_consensus.cc:2229] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:19:52.995613 12451 raft_consensus.cc:2258] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.995680 12453 raft_consensus.cc:2258] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.995775 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 11904
I20260809 06:19:52.995921 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet e586a32d183f4ed98486b1ee5758a337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:19:52 UTC) from {username='oryx', principal='oryx/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:47608
I20260809 06:19:52.996271 12452 tablet_replica.cc:330] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: stopping tablet replica
I20260809 06:19:52.996461 12452 raft_consensus.cc:2229] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:19:52.996727 12452 raft_consensus.cc:2258] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:19:52.997325 12453 ts_tablet_manager.cc:1905] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.998183 12452 ts_tablet_manager.cc:1905] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:52.999222 12451 ts_tablet_manager.cc:1905] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:19:53.004199 12453 ts_tablet_manager.cc:1918] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:53.004340 12453 log.cc:1196] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/e586a32d183f4ed98486b1ee5758a337
I20260809 06:19:53.004772 12453 ts_tablet_manager.cc:1939] T e586a32d183f4ed98486b1ee5758a337 P daaa3eae3ccc4105b3551349d7e4e336: Deleting consensus metadata
I20260809 06:19:53.005122 12452 ts_tablet_manager.cc:1918] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:53.005265 12452 log.cc:1196] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/e586a32d183f4ed98486b1ee5758a337
I20260809 06:19:53.005728 12452 ts_tablet_manager.cc:1939] T e586a32d183f4ed98486b1ee5758a337 P 4caa226cb74d41daa93c098cb1eaeb41: Deleting consensus metadata
I20260809 06:19:53.005847 11810 catalog_manager.cc:4841] TS daaa3eae3ccc4105b3551349d7e4e336 (127.3.137.67:33331): tablet e586a32d183f4ed98486b1ee5758a337 (table test-table [id=8bfb7f728e1247f2827002706cd04d52]) successfully deleted
I20260809 06:19:53.006327 12451 ts_tablet_manager.cc:1918] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:19:53.006528 12451 log.cc:1196] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/e586a32d183f4ed98486b1ee5758a337
I20260809 06:19:53.006930 12451 ts_tablet_manager.cc:1939] T e586a32d183f4ed98486b1ee5758a337 P fd3b1eb34c20483993c4cc5dedc58de9: Deleting consensus metadata
I20260809 06:19:53.006918 11811 catalog_manager.cc:4841] TS 4caa226cb74d41daa93c098cb1eaeb41 (127.3.137.66:40399): tablet e586a32d183f4ed98486b1ee5758a337 (table test-table [id=8bfb7f728e1247f2827002706cd04d52]) successfully deleted
I20260809 06:19:53.007899 11810 catalog_manager.cc:4841] TS fd3b1eb34c20483993c4cc5dedc58de9 (127.3.137.65:39019): tablet e586a32d183f4ed98486b1ee5758a337 (table test-table [id=8bfb7f728e1247f2827002706cd04d52]) successfully deleted
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.65@KRBTEST.COM
I20260809 06:19:53.016016 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:52.997318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.3.137.65:39019)
0809 06:19:52.997878 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:52.997886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:52.998331 (+ 445us) server_negotiation.cc:407] Connection header received
0809 06:19:52.998612 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:52.998620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:52.998707 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:52.998832 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:53.000895 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.002019 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.004127 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.004816 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.004939 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:53.010295 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:53.010311 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:53.010327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:53.010358 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:53.013208 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.014024 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.014032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.014034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.014160 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.014554 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.014557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.014559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.014925 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:53.015239 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:53.015559 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 06:19:53.015722 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260809 06:19:53.016652 12047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:53.175518 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 11904
I20260809 06:19:53.206516 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12059
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.66@KRBTEST.COM
I20260809 06:19:53.221592 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:53.207396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.3.137.66:40399)
0809 06:19:53.207583 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:53.207592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:53.207645 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:19:53.207877 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:53.207887 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:53.207952 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:53.208060 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:53.209339 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.210046 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.211522 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.211924 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.212060 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:53.216192 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:53.216205 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:53.216221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:53.216259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:53.218902 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.219616 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.219623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.219625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.219816 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.220186 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.220190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.220193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.220631 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:53.220895 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:53.221160 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 06:19:53.221345 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:19:53.222291 12201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:53.353545 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12059
I20260809 06:19:53.379508 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12211
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[11777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.137.67@KRBTEST.COM
I20260809 06:19:53.395689 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:53.380341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.3.137.67:33331)
0809 06:19:53.380868 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:53.380876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:53.380898 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:19:53.380952 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:53.380956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:53.381028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:53.381149 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:53.382515 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.383331 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.384610 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:53.385017 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:53.385167 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:53.389691 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:53.389706 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:53.389724 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:53.389759 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:53.392846 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.393530 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.393538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.393540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.393682 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:53.394072 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:53.394078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:53.394081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:53.394446 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:53.394767 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:53.395109 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 06:19:53.395358 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260809 06:19:53.396399 12351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:53.495102 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12211
I20260809 06:19:53.515173 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 11792
I20260809 06:19:53.516408 11871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:53.635088 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 11792
2026-08-09T06:19:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7228 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[12511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[12511](info): set up 2 sockets
Aug 09 06:19:53 dist-test-slave-2w3w krb5kdc[12511](info): commencing operation
krb5kdc: starting...
W20260809 06:19:54.783437 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:54 dist-test-slave-2w3w krb5kdc[12511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:55.054908 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37317
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:55.397609 12524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:55.397907 12524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:55.398075 12524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:55.405510 12524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:55.405627 12524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:55.405733 12524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:55.405848 12524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:55.405939 12524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:55.416081 12524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:55.417429 12524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:55.418766 12524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:55.431308 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:55.431314 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:55.431432 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:55.431876 12524 server_base.cc:1034] running on GCE node
I20260809 06:19:55.432830 12524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:55.434334 12524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:55.435604 12524 hybrid_clock.cc:648] HybridClock initialized: now 1786256395435545 us; error 49 us; skew 500 ppm
I20260809 06:19:55.436010 12524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T06:19:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1875 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:19:55 dist-test-slave-2w3w krb5kdc[12540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:19:55 dist-test-slave-2w3w krb5kdc[12540](info): set up 2 sockets
Aug 09 06:19:55 dist-test-slave-2w3w krb5kdc[12540](info): commencing operation
krb5kdc: starting...
W20260809 06:19:56.670447 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:19:56 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:19:56.915452 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37999
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:57.243420 12554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:57.243690 12554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:57.243883 12554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:57.251220 12554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:57.251344 12554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:57.251448 12554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:57.251567 12554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:57.251631 12554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:19:57.261260 12554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:57.262480 12554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:57.263741 12554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:57.275390 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:57.275274 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:57.275476 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:57.276427 12554 server_base.cc:1034] running on GCE node
I20260809 06:19:57.277149 12554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:57.278445 12554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:57.279675 12554 hybrid_clock.cc:648] HybridClock initialized: now 1786256397279630 us; error 34 us; skew 500 ppm
I20260809 06:19:57.280014 12554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:57 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:57.283106 12554 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:19:57.284567 12554 webserver.cc:460] Webserver started at http://127.3.137.126:39611/ using document root <none> and password file <none>
I20260809 06:19:57.285447 12554 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:57.285604 12554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:57.285910 12554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:57.288847 12554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "e86e7cc56456464cbd0944ce501134e6"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "e10689d46ccd5bad5ef08c43a6dac49a"
server_key_iv: "21b0cdd5d256fa60108a663b31c7c260"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.289647 12554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "e86e7cc56456464cbd0944ce501134e6"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "e10689d46ccd5bad5ef08c43a6dac49a"
server_key_iv: "21b0cdd5d256fa60108a663b31c7c260"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.294798 12554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:19:57.298269 12571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:57.299641 12554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:57.299853 12554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "e86e7cc56456464cbd0944ce501134e6"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "e10689d46ccd5bad5ef08c43a6dac49a"
server_key_iv: "21b0cdd5d256fa60108a663b31c7c260"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.300047 12554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:57.320847 12554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:57.325095 12554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:57.325379 12554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:57.343586 12554 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:37999
I20260809 06:19:57.343626 12624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:37999 every 8 connection(s)
I20260809 06:19:57.344861 12554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:19:57.348472 12625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:19:57.349515 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12554
I20260809 06:19:57.349813 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:19:57.350733 3621 external_mini_cluster.cc:1428] Setting key cb2ca3fe46e7718774daa6698cf0eeb0
I20260809 06:19:57.361114 12625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Bootstrap starting.
I20260809 06:19:57.364946 12625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Neither blocks nor log segments found. Creating new log.
Aug 09 06:19:57 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:57.365896 12625 log.cc:824] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:19:57.368870 12625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: No bootstrap required, opened a new log
I20260809 06:19:57.373313 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:57.352820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.3.137.126:37999)
0809 06:19:57.353560 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:57.353571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:57.353611 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:19:57.355318 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:57.355339 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:57.355690 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:57.356079 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:57.357585 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.359129 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.360465 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.361138 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.361341 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:57.366073 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:57.366082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:57.366095 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:57.366131 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:57.369316 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:57.370009 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:57.370021 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:57.370027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:57.370193 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:57.370601 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:57.370608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:57.370616 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:57.371194 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:57.371534 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:57.371963 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 06:19:57.372372 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":345,"threads_started":1}
I20260809 06:19:57.376643 12625 raft_consensus.cc:348] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:19:57.376969 12625 raft_consensus.cc:374] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:19:57.377084 12625 raft_consensus.cc:729] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86e7cc56456464cbd0944ce501134e6, State: Initialized, Role: FOLLOWER
I20260809 06:19:57.377604 12625 consensus_queue.cc:260] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:19:57.377836 12625 raft_consensus.cc:388] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:19:57.377998 12625 raft_consensus.cc:482] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:19:57.378186 12625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:19:57.380673 12625 raft_consensus.cc:504] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:19:57.381161 12625 leader_election.cc:302] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e86e7cc56456464cbd0944ce501134e6; no voters:
I20260809 06:19:57.381723 12625 leader_election.cc:288] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:19:57.381897 12630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:19:57.382618 12630 raft_consensus.cc:686] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 LEADER]: Becoming Leader. State: Replica: e86e7cc56456464cbd0944ce501134e6, State: Running, Role: LEADER
I20260809 06:19:57.383283 12630 consensus_queue.cc:237] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:19:57.383761 12625 sys_catalog.cc:564] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:19:57.386221 12631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e86e7cc56456464cbd0944ce501134e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } } }
I20260809 06:19:57.386654 12631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:57.392062 12632 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e86e7cc56456464cbd0944ce501134e6. Latest consensus state: current_term: 1 leader_uuid: "e86e7cc56456464cbd0944ce501134e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } } }
I20260809 06:19:57.392311 12637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:19:57.392444 12632 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:19:57.396832 12637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:19:57.405023 12637 catalog_manager.cc:1347] Generated new cluster ID: 898fd59827b2424eb9e17ad775d71535
I20260809 06:19:57.405138 12637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:19:57.432077 12637 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:19:57.433410 12637 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:19:57.443732 12637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Generated new TSK 0
I20260809 06:19:57.444583 12637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:19:57.508463 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:57.838311 12651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:57.838574 12651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:57.838759 12651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:57.845902 12651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:57.846011 12651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:57.846215 12651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:19:57.855938 12651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:57.857205 12651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:57.858528 12651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:57.871582 12657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:57.871523 12659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:57.871624 12656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:57.872634 12651 server_base.cc:1034] running on GCE node
I20260809 06:19:57.873435 12651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:57.874426 12651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:57.875653 12651 hybrid_clock.cc:648] HybridClock initialized: now 1786256397875578 us; error 61 us; skew 500 ppm
I20260809 06:19:57.875978 12651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:57 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:57.879187 12651 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:19:57.880661 12651 webserver.cc:460] Webserver started at http://127.3.137.65:41087/ using document root <none> and password file <none>
I20260809 06:19:57.881335 12651 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:57.881484 12651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:57.881806 12651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:57.884982 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "12d3fc4338ae4a1da3b0eeb6884543cf"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "13d24f8a7a8f0af66bea40d53bba7b36"
server_key_iv: "3aa6f36ea6723d0007326740d08ac22b"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.885784 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "12d3fc4338ae4a1da3b0eeb6884543cf"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "13d24f8a7a8f0af66bea40d53bba7b36"
server_key_iv: "3aa6f36ea6723d0007326740d08ac22b"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.891122 12651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:19:57.894519 12666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:57.895866 12651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:19:57.896039 12651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "12d3fc4338ae4a1da3b0eeb6884543cf"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "13d24f8a7a8f0af66bea40d53bba7b36"
server_key_iv: "3aa6f36ea6723d0007326740d08ac22b"
server_key_version: "encryptionkey@0"
I20260809 06:19:57.896248 12651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:57.916450 12651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:57.920506 12651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:57.920814 12651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:57.921846 12651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:57.923216 12651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:57.923322 12651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:57.923480 12651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:57.923554 12651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:57.957718 12651 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:46647
I20260809 06:19:57.957779 12790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:46647 every 8 connection(s)
I20260809 06:19:57.959157 12651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:19:57.962477 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12651
I20260809 06:19:57.962709 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:19:57.963723 3621 external_mini_cluster.cc:1428] Setting key 39f865a050a520dc41c06aff1190511c
Aug 09 06:19:57 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:57.983608 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:57.961885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:38125 (local address 127.3.137.126:37999)
0809 06:19:57.962450 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:57.962456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:57.963663 (+ 1207us) server_negotiation.cc:407] Connection header received
0809 06:19:57.966074 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:57.966078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:57.966147 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:57.966261 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:57.968913 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.969674 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.972347 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.972839 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.972954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:57.977511 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:57.977516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:57.977520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:57.977557 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:57.980171 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:57.981159 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:57.981163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:57.981165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:57.981288 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:57.981804 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:57.981807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:57.981809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:57.982070 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:57.982181 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:57.983151 (+ 970us) server_negotiation.cc:299] Negotiation successful
0809 06:19:57.983346 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":271,"threads_started":1}
I20260809 06:19:57.985245 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:57.962352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:38125)
0809 06:19:57.963440 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:57.963506 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:57.965663 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:57.966559 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:57.966572 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:57.967713 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:57.968679 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:57.968710 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.969855 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.969862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:57.972160 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:57.972179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:57.973101 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:57.973112 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:57.973476 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:57.974323 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:57.974380 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:57.977234 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:57.980330 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:57.980343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:57.980365 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:57.981029 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:57.981399 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:57.981407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:57.981413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:57.981680 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:57.982209 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:57.982218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:57.982697 (+ 479us) client_negotiation.cc:769] Sending connection context
0809 06:19:57.983047 (+ 350us) client_negotiation.cc:240] Negotiation successful
0809 06:19:57.983661 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":450,"threads_started":1}
I20260809 06:19:57.986956 12791 heartbeater.cc:347] Connected to a master server at 127.3.137.126:37999
I20260809 06:19:57.987327 12791 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:57.988106 12791 heartbeater.cc:511] Master 127.3.137.126:37999 requested a full tablet report, sending...
I20260809 06:19:57.990483 12588 ts_manager.cc:194] Registered new tserver with Master: 12d3fc4338ae4a1da3b0eeb6884543cf (127.3.137.65:46647)
I20260809 06:19:57.992022 12588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:38125
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:19:58.025808 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:58.356982 12797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:58.357293 12797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:58.357460 12797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:58.365154 12797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:58.365310 12797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:58.365511 12797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:19:58.374799 12797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:58.375982 12797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:58.377197 12797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:58.390489 12803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:58.390420 12806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:58.390475 12804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:58.391706 12797 server_base.cc:1034] running on GCE node
I20260809 06:19:58.392313 12797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:58.393210 12797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:58.394443 12797 hybrid_clock.cc:648] HybridClock initialized: now 1786256398394402 us; error 49 us; skew 500 ppm
I20260809 06:19:58.394784 12797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:58 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:58.398000 12797 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:19:58.399595 12797 webserver.cc:460] Webserver started at http://127.3.137.66:35173/ using document root <none> and password file <none>
I20260809 06:19:58.400261 12797 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:58.400377 12797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:58.400694 12797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:58.403674 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "cf62dc7df25b402c8493be0707536858"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "df8fdf3def03a0396dab51a625bcc57f"
server_key_iv: "24b7d5d9d9e63641aa4f6f2f5c0da403"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.404516 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "cf62dc7df25b402c8493be0707536858"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "df8fdf3def03a0396dab51a625bcc57f"
server_key_iv: "24b7d5d9d9e63641aa4f6f2f5c0da403"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.409508 12797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:19:58.412774 12813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.414077 12797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:19:58.414211 12797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "cf62dc7df25b402c8493be0707536858"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "df8fdf3def03a0396dab51a625bcc57f"
server_key_iv: "24b7d5d9d9e63641aa4f6f2f5c0da403"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.414388 12797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:58.435669 12797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:58.440289 12797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:58.440618 12797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:58.441568 12797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:58.442922 12797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:58.443013 12797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.443107 12797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:58.443150 12797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.477406 12797 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:42901
I20260809 06:19:58.477490 12934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:42901 every 8 connection(s)
I20260809 06:19:58.478559 12797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:19:58.480186 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12797
I20260809 06:19:58.480365 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:19:58.480871 3621 external_mini_cluster.cc:1428] Setting key f5a5f517c5298a1347817b8c0f96ef55
Aug 09 06:19:58 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:58.498903 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:58.481185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:43277 (local address 127.3.137.126:37999)
0809 06:19:58.481406 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:58.481414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:58.482586 (+ 1172us) server_negotiation.cc:407] Connection header received
0809 06:19:58.484631 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:58.484636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:58.484731 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:58.484842 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:58.486553 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.487337 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.488902 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.489394 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.489523 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:58.493070 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:58.493074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:58.493078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:58.493110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:58.495858 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:58.496571 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:58.496578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:58.496582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:58.496715 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:58.497144 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:58.497150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:58.497154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:58.497427 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:58.497583 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:58.498330 (+ 747us) server_negotiation.cc:299] Negotiation successful
0809 06:19:58.498561 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:19:58.499997 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:58.481678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:43277)
0809 06:19:58.482429 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:58.482470 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:58.484350 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:58.485138 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:58.485146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:58.485636 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:58.486358 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:58.486387 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.487492 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.487497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:58.488754 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:58.488767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.489653 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.489660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:58.489880 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:58.490631 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:58.490653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:58.492851 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:58.496000 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:58.496009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:58.496024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:58.496451 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:58.496830 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:58.496835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:58.496839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:58.497026 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:58.497567 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:58.497574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:58.497933 (+ 359us) client_negotiation.cc:769] Sending connection context
0809 06:19:58.498206 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 06:19:58.498726 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":305,"threads_started":1}
I20260809 06:19:58.501442 12935 heartbeater.cc:347] Connected to a master server at 127.3.137.126:37999
I20260809 06:19:58.501761 12935 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:58.502455 12935 heartbeater.cc:511] Master 127.3.137.126:37999 requested a full tablet report, sending...
I20260809 06:19:58.504150 12588 ts_manager.cc:194] Registered new tserver with Master: cf62dc7df25b402c8493be0707536858 (127.3.137.66:42901)
I20260809 06:19:58.505067 12588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:43277
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:19:58.538069 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:58.866467 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:58.866727 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:58.866822 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:58.874125 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:58.874228 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:58.874338 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:19:58.883947 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:19:58.885370 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:19:58.886631 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:19:58.899597 12945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:58.899601 12946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:19:58.899886 12948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:19:58.900353 12940 server_base.cc:1034] running on GCE node
I20260809 06:19:58.900955 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:19:58.901952 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:19:58.903160 12940 hybrid_clock.cc:648] HybridClock initialized: now 1786256398903098 us; error 54 us; skew 500 ppm
I20260809 06:19:58.903431 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:19:58 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:19:58.906677 12940 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:19:58.908141 12940 webserver.cc:460] Webserver started at http://127.3.137.67:35657/ using document root <none> and password file <none>
I20260809 06:19:58.908850 12940 fs_manager.cc:359] Metadata directory not provided
I20260809 06:19:58.908954 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:19:58.909221 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:19:58.912189 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "956f658e79ed44d78e7b317b357163e0"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "f5e65a16e9568696f296f780bd2e5289"
server_key_iv: "d508e27d86f93bc35e65e85efd582488"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.912937 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "956f658e79ed44d78e7b317b357163e0"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "f5e65a16e9568696f296f780bd2e5289"
server_key_iv: "d508e27d86f93bc35e65e85efd582488"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.918334 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:19:58.921911 12955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.923292 12940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:19:58.923430 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "956f658e79ed44d78e7b317b357163e0"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "f5e65a16e9568696f296f780bd2e5289"
server_key_iv: "d508e27d86f93bc35e65e85efd582488"
server_key_version: "encryptionkey@0"
I20260809 06:19:58.923593 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:19:58.936395 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:19:58.940907 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:19:58.941190 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:19:58.942236 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:19:58.943583 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:19:58.943660 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.943743 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:19:58.943799 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:19:58.979249 12940 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33593
I20260809 06:19:58.979292 13075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33593 every 8 connection(s)
I20260809 06:19:58.980655 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:19:58.982020 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 12940
I20260809 06:19:58.982215 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:19:58.982800 3621 external_mini_cluster.cc:1428] Setting key dfcc703cc37cacbcd8bcddaa970478a3
Aug 09 06:19:58 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256398, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:19:58.995136 12791 heartbeater.cc:503] Master 127.3.137.126:37999 was elected leader, sending a full tablet report...
I20260809 06:19:59.001240 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:58.983523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:59105 (local address 127.3.137.126:37999)
0809 06:19:58.983739 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:58.983746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:58.985145 (+ 1399us) server_negotiation.cc:407] Connection header received
0809 06:19:58.987286 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:58.987290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:58.987355 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:58.987438 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:58.988865 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.989696 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.991283 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.991687 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.991781 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:58.995097 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:58.995102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:58.995105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:58.995137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:58.998209 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:58.999009 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:58.999013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:58.999015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:58.999137 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:58.999566 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:58.999570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:58.999574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:58.999840 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:59.000007 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:59.000679 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 06:19:59.000891 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:19:59.002177 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:58.984073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:59105)
0809 06:19:58.984983 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 06:19:58.985044 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 06:19:58.987044 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:19:58.987652 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:19:58.987659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:19:58.988137 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:19:58.988705 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:58.988722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.989862 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.989865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:58.991149 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:19:58.991161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:19:58.991869 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:19:58.991873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:19:58.992060 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:58.992708 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:19:58.992728 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:19:58.994866 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:19:58.998346 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:58.998355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:58.998370 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:58.998848 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:58.999252 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:19:58.999257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:19:58.999261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:19:58.999442 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:19:58.999992 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:19:58.999999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:19:59.000315 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 06:19:59.000565 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:19:59.001084 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":347,"threads_started":1}
I20260809 06:19:59.003459 13076 heartbeater.cc:347] Connected to a master server at 127.3.137.126:37999
I20260809 06:19:59.003822 13076 heartbeater.cc:464] Registering TS with master...
I20260809 06:19:59.004683 13076 heartbeater.cc:511] Master 127.3.137.126:37999 requested a full tablet report, sending...
I20260809 06:19:59.006050 12588 ts_manager.cc:194] Registered new tserver with Master: 956f658e79ed44d78e7b317b357163e0 (127.3.137.67:33593)
I20260809 06:19:59.006847 12588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:59105
I20260809 06:19:59.010751 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
Entry for principal oryx/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab.
WARNING: no policy specified for oryx/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
Entry for principal oryx/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab.
WARNING: no policy specified for oryx/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
Entry for principal oryx/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab.
I20260809 06:19:59.165863 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12651
Aug 09 06:19:59 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:19:59.183468 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:59.166996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.3.137.65:46647)
0809 06:19:59.167594 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:59.167602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:59.168184 (+ 582us) server_negotiation.cc:407] Connection header received
0809 06:19:59.168357 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:59.168364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:59.168584 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:59.168731 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:59.170247 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.171325 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.172645 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.173193 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.173289 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:59.177821 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:59.177832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:59.177846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:59.177877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:59.180606 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.181337 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.181345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.181348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.181475 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.181855 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.181860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.181863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.182199 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:59.182515 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:59.182810 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 06:19:59.183122 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":286,"threads_started":1}
I20260809 06:19:59.184854 12785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:59.269375 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12651
I20260809 06:19:59.287249 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12797
Aug 09 06:19:59 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:19:59.303781 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:59.288138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.3.137.66:42901)
0809 06:19:59.288735 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:59.288742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:59.288767 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:19:59.288855 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:59.288865 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:59.289092 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:59.289260 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:59.290605 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.291745 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.293046 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.293581 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.293727 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:59.298405 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:59.298415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:59.298428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:59.298469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:59.301151 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.301788 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.301794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.301796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.301920 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.302263 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.302266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.302267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.302586 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:59.302861 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:59.303271 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 06:19:59.303492 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":336,"threads_started":1}
I20260809 06:19:59.305063 12930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:59.393437 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12797
I20260809 06:19:59.410538 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12940
Aug 09 06:19:59 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:19:59.427588 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:19:59.411588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.3.137.67:33593)
0809 06:19:59.412279 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0809 06:19:59.412286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:19:59.412314 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:19:59.412372 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:19:59.412379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:19:59.412586 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:19:59.412802 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:19:59.414181 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.415351 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.416622 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:19:59.417171 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:19:59.417319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:19:59.421857 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:19:59.421868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:19:59.421882 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:19:59.421915 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:19:59.424863 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.425483 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.425489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.425492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.425622 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:19:59.425970 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:19:59.425973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:19:59.425975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:19:59.426312 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:19:59.426545 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 06:19:59.427003 (+ 458us) server_negotiation.cc:299] Negotiation successful
0809 06:19:59.427277 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":369,"threads_started":1}
I20260809 06:19:59.428985 13071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:59.513639 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12940
I20260809 06:19:59.531226 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 12554
I20260809 06:19:59.532490 12620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:19:59.637079 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 12554
I20260809 06:19:59.659762 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37999
--webserver_interface=127.3.137.126
--webserver_port=39611
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:19:59.987136 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:19:59.987438 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:19:59.987625 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:19:59.994910 13093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:19:59.995100 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:19:59.995205 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:19:59.995321 13093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:19:59.995407 13093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:20:00.005517 13093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:00.006932 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:00.008319 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:00.020593 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.020653 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.020676 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:00.021555 13093 server_base.cc:1034] running on GCE node
I20260809 06:20:00.022403 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:00.023944 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:00.025183 13093 hybrid_clock.cc:648] HybridClock initialized: now 1786256400025142 us; error 54 us; skew 500 ppm
I20260809 06:20:00.025535 13093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:00 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:00.028736 13093 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:20:00.030349 13093 webserver.cc:460] Webserver started at http://127.3.137.126:39611/ using document root <none> and password file <none>
I20260809 06:20:00.031133 13093 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:00.031299 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:00.036895 13093 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 06:20:00.039783 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:00.041057 13093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:20:00.041245 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "e86e7cc56456464cbd0944ce501134e6"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "e10689d46ccd5bad5ef08c43a6dac49a"
server_key_iv: "21b0cdd5d256fa60108a663b31c7c260"
server_key_version: "encryptionkey@0"
I20260809 06:20:00.042233 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:00.063673 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:00.067874 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:00.068166 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:00.087569 13093 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:37999
I20260809 06:20:00.087584 13162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:37999 every 8 connection(s)
I20260809 06:20:00.089156 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:20:00.093878 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13093
I20260809 06:20:00.094985 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:46647
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:37999
--webserver_port=41087
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:20:00.097124 13163 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:20:00.102277 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Bootstrap starting.
I20260809 06:20:00.122359 13163 log.cc:824] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:00.131397 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:20:00.131951 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Bootstrap complete.
I20260809 06:20:00.145555 13163 raft_consensus.cc:348] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:20:00.146601 13163 raft_consensus.cc:729] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86e7cc56456464cbd0944ce501134e6, State: Initialized, Role: FOLLOWER
I20260809 06:20:00.147383 13163 consensus_queue.cc:260] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:20:00.147768 13163 raft_consensus.cc:388] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:20:00.147938 13163 raft_consensus.cc:482] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:20:00.148154 13163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:20:00.152297 13163 raft_consensus.cc:504] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:20:00.152894 13163 leader_election.cc:302] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e86e7cc56456464cbd0944ce501134e6; no voters:
I20260809 06:20:00.153551 13163 leader_election.cc:288] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:20:00.153699 13167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:20:00.155176 13167 raft_consensus.cc:686] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [term 2 LEADER]: Becoming Leader. State: Replica: e86e7cc56456464cbd0944ce501134e6, State: Running, Role: LEADER
I20260809 06:20:00.155802 13167 consensus_queue.cc:237] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } }
I20260809 06:20:00.156083 13163 sys_catalog.cc:564] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:20:00.158479 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e86e7cc56456464cbd0944ce501134e6. Latest consensus state: current_term: 2 leader_uuid: "e86e7cc56456464cbd0944ce501134e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } } }
I20260809 06:20:00.158859 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:00.159447 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e86e7cc56456464cbd0944ce501134e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86e7cc56456464cbd0944ce501134e6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 37999 } } }
I20260809 06:20:00.159770 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:00.169498 13176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:20:00.175340 13176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:20:00.176855 13176 catalog_manager.cc:1259] Loaded cluster ID: 898fd59827b2424eb9e17ad775d71535
I20260809 06:20:00.176957 13176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:20:00.181123 13176 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:20:00.182628 13176 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e86e7cc56456464cbd0944ce501134e6: Loaded TSK: 0
I20260809 06:20:00.184083 13176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:20:00.445350 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:00.445681 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:00.445909 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:00.453222 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:00.453401 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:00.453603 13165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:20:00.463392 13165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.65.keytab
--principal=oryx/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:46647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=41087
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:00.464736 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:00.466002 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:00.480713 13192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.481313 13193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.482656 13195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:00.482661 13165 server_base.cc:1034] running on GCE node
I20260809 06:20:00.483379 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:00.484303 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:00.485540 13165 hybrid_clock.cc:648] HybridClock initialized: now 1786256400485495 us; error 38 us; skew 500 ppm
I20260809 06:20:00.485852 13165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:00 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256400, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:00.491992 13165 init.cc:377] Logged in from keytab as oryx/127.3.137.65@KRBTEST.COM (short username oryx)
I20260809 06:20:00.493607 13165 webserver.cc:460] Webserver started at http://127.3.137.65:41087/ using document root <none> and password file <none>
I20260809 06:20:00.494309 13165 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:00.494463 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:00.499851 13165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:20:00.502569 13202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:00.504174 13165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:20:00.504380 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "12d3fc4338ae4a1da3b0eeb6884543cf"
format_stamp: "Formatted at 2026-08-09 06:19:57 on dist-test-slave-2w3w"
server_key: "13d24f8a7a8f0af66bea40d53bba7b36"
server_key_iv: "3aa6f36ea6723d0007326740d08ac22b"
server_key_version: "encryptionkey@0"
I20260809 06:20:00.505376 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:00.535503 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:00.539341 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:00.539616 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:00.540580 13165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:00.541879 13165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:00.542004 13165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:00.542142 13165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:00.542244 13165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:00.576571 13165 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:46647
I20260809 06:20:00.576660 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:46647 every 8 connection(s)
I20260809 06:20:00.577973 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:20:00.583401 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13165
I20260809 06:20:00.584034 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:42901
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:37999
--webserver_port=35173
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:20:00 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:00 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:20:00.916255 13324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:00.916543 13324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:00.916695 13324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:00.924202 13324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:00.924324 13324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:00.924537 13324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:20:00.934556 13324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.66.keytab
--principal=oryx/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:42901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=35173
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:00.936010 13324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:00.937786 13324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:00.954197 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.954722 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:00.955577 13335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:00.960955 13324 server_base.cc:1034] running on GCE node
I20260809 06:20:00.962179 13324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:00.964131 13324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:00.965483 13324 hybrid_clock.cc:648] HybridClock initialized: now 1786256400965393 us; error 59 us; skew 500 ppm
I20260809 06:20:00.966082 13324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:00 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256400, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:00.972609 13324 init.cc:377] Logged in from keytab as oryx/127.3.137.66@KRBTEST.COM (short username oryx)
I20260809 06:20:00.975428 13324 webserver.cc:460] Webserver started at http://127.3.137.66:35173/ using document root <none> and password file <none>
I20260809 06:20:00.976938 13324 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:00.977186 13324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:00.988590 13324 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.005s
I20260809 06:20:00.993590 13342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:00.996013 13324 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260809 06:20:00.996284 13324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "cf62dc7df25b402c8493be0707536858"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "df8fdf3def03a0396dab51a625bcc57f"
server_key_iv: "24b7d5d9d9e63641aa4f6f2f5c0da403"
server_key_version: "encryptionkey@0"
I20260809 06:20:00.998360 13324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:01.055815 13324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:01.063977 13324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:01.064491 13324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:01.066237 13324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:01.068701 13324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:01.068879 13324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:01.069033 13324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:01.069146 13324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:01.120267 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:42901 every 8 connection(s)
I20260809 06:20:01.120275 13324 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:42901
I20260809 06:20:01.122918 13324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:20:01.131472 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13324
I20260809 06:20:01.132503 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:33593
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:37999
--webserver_port=35657
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:20:01.771131 13464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:01.771457 13464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:01.771625 13464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:01.778935 13464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:01.779142 13464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:01.779352 13464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:20:01.789232 13464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/krb5kdc/oryx_127.3.137.67.keytab
--principal=oryx/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:33593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=35657
--tserver_master_addrs=127.3.137.126:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:01.790614 13464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:01.792155 13464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:01.793222 13322 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.202s user 0.005s sys 0.008s
I20260809 06:20:01.795274 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:00.580798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:39653)
0809 06:20:00.581722 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:00.581761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:00.583504 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:00.584943 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:00.584953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:00.585522 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:00.586043 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:00.586060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:00.587702 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:00.587707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:00.588935 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:00.588958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:00.590029 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:00.590036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:00.590297 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:00.590942 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:00.591012 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:01.794149 (+1203137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":546,"threads_started":1}
W20260809 06:20:01.796173 13321 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:00.580811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:39653 (local address 127.3.137.126:37999)
0809 06:20:00.581569 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:00.581610 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:00.581888 (+ 278us) server_negotiation.cc:407] Connection header received
0809 06:20:00.584043 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:00.584055 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:00.584441 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:00.584781 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:00.586240 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:00.587545 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:00.589100 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:00.589785 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:00.589998 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.794879 (+1204881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:39653: BlockingRecv error: recv got EOF from 127.3.137.65:39653 (error 108)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":248,"threads_started":1}
W20260809 06:20:01.796787 13318 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:37999 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.137.126:37999: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:20:01.812367 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:01.814488 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:01.797540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:57509)
0809 06:20:01.798125 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:01.798147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:01.798288 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:01.798795 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:01.798800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:01.799287 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:01.799502 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.799514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.801615 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.801625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.804045 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.804068 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.805533 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.805543 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.805784 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.806621 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:01.806657 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:01.814015 (+ 7358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":297}
W20260809 06:20:01.815554 13321 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:01.797783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:57509 (local address 127.3.137.126:37999)
0809 06:20:01.798041 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:01.798052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:01.798245 (+ 193us) server_negotiation.cc:407] Connection header received
0809 06:20:01.798456 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:01.798466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:01.798581 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:01.798740 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:01.799673 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.801364 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.804341 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.805263 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.806218 (+ 955us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.815206 (+ 8988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:57509: BlockingRecv error: recv got EOF from 127.3.137.65:57509 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 06:20:01.816279 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:01.822495 13473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:01.823216 13464 server_base.cc:1034] running on GCE node
I20260809 06:20:01.824280 13464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:01.825646 13464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:01.827167 13464 hybrid_clock.cc:648] HybridClock initialized: now 1786256401827062 us; error 91 us; skew 500 ppm
I20260809 06:20:01.827579 13464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260809 06:20:01.829028 13321 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:01.815905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:47669 (local address 127.3.137.126:37999)
0809 06:20:01.816194 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:01.816200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:01.816647 (+ 447us) server_negotiation.cc:407] Connection header received
0809 06:20:01.816719 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:01.816727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:01.816857 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:01.816965 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:01.818156 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.819728 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.822094 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.823005 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.823139 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.828803 (+ 5664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:47669: BlockingRecv error: recv got EOF from 127.3.137.65:47669 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260809 06:20:01.829735 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:01.815430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:47669)
0809 06:20:01.816359 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:01.816385 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:01.816523 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:01.817187 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:01.817194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:01.817685 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:01.817935 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.817955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.819969 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.819977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.821920 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.821940 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.823373 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.823380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.823630 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.824353 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:01.824383 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:01.829449 (+ 5066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":792}
W20260809 06:20:01.830135 13318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256401, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:01.832145 13464 init.cc:377] Logged in from keytab as oryx/127.3.137.67@KRBTEST.COM (short username oryx)
I20260809 06:20:01.833923 13464 webserver.cc:460] Webserver started at http://127.3.137.67:35657/ using document root <none> and password file <none>
I20260809 06:20:01.834676 13464 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:01.834863 13464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:01.841217 13464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 06:20:01.844542 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:01.845974 13464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:20:01.846163 13464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "956f658e79ed44d78e7b317b357163e0"
format_stamp: "Formatted at 2026-08-09 06:19:58 on dist-test-slave-2w3w"
server_key: "f5e65a16e9568696f296f780bd2e5289"
server_key_iv: "d508e27d86f93bc35e65e85efd582488"
server_key_version: "encryptionkey@0"
I20260809 06:20:01.847258 13464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:01.859467 13464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:01.863924 13464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:01.864195 13464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:01.865293 13464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:01.866686 13464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:01.866815 13464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:01.866952 13464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:01.867071 13464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 06:20:01.910876 13464 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33593
I20260809 06:20:01.910939 13599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33593 every 8 connection(s)
I20260809 06:20:01.912664 13464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:20:01.913270 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13464
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:01 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:01.969017 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:01.914683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.3.137.126:37999)
0809 06:20:01.914939 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:01.914945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:01.945297 (+ 30352us) server_negotiation.cc:407] Connection header received
0809 06:20:01.945622 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:01.945632 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:01.945768 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:01.945936 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:01.948634 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.950427 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.952368 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.952868 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.953018 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.957443 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:01.957452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:01.957467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:01.957513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:01.963769 (+ 6256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:01.964847 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:01.964862 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:01.964878 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:01.965236 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:01.965754 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:01.965762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:01.965768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:01.966767 (+ 999us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:01.967254 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:01.967817 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 06:20:01.968416 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111}
W20260809 06:20:02.382825 13463 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.247s user 0.001s sys 0.013s
W20260809 06:20:02.384579 13462 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:01.126217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:60927 (local address 127.3.137.126:37999)
0809 06:20:01.126727 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:01.126731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:01.128135 (+ 1404us) server_negotiation.cc:407] Connection header received
0809 06:20:01.130015 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:01.130021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:01.130123 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:01.130252 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:01.131873 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.132680 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.134340 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.134771 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.134907 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.384339 (+1249432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:60927: BlockingRecv error: recv got EOF from 127.3.137.66:60927 (error 108)
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
I20260809 06:20:02.386015 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:01.127093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:60927)
0809 06:20:01.127973 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:01.128010 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:01.129748 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:01.130499 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:01.130507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:01.131087 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:01.131678 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.131708 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.132898 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.132904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.134175 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.134187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.135043 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.135054 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.135291 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.135968 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:01.135990 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:02.384214 (+1248224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":384,"threads_started":1}
W20260809 06:20:02.386996 13460 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:37999 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.137.126:37999: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:02.400434 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:02.387634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:46149)
0809 06:20:02.387927 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:02.387955 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:02.388136 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:02.388499 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:02.388507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:02.389199 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:02.389530 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.389548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.390619 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.390628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.392989 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.393011 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.394072 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.394082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.394377 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.395264 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:02.395300 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:02.400223 (+ 4923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 06:20:02.400975 13462 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:02.387773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:46149 (local address 127.3.137.126:37999)
0809 06:20:02.387914 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:02.387918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:02.388030 (+ 112us) server_negotiation.cc:407] Connection header received
0809 06:20:02.388270 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:02.388275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:02.388339 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:02.388418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:02.389696 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.390453 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.393173 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.393804 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.393930 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.400629 (+ 6699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:46149: BlockingRecv error: recv got EOF from 127.3.137.66:46149 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:20:02.412636 13462 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:02.401487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:38087 (local address 127.3.137.126:37999)
0809 06:20:02.401725 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:02.401735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:02.401762 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:20:02.401834 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:02.401842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:02.401975 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:02.402124 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:02.402740 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.404462 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.406272 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.406916 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.407084 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.412430 (+ 5346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:38087: BlockingRecv error: recv got EOF from 127.3.137.66:38087 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:20:02.412694 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:02.401322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:38087)
0809 06:20:02.401553 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:02.401571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:02.401664 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:02.402143 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:02.402145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:02.402441 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:02.402591 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.402598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.404660 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.404667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.406106 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.406125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.407103 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.407109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.407310 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.407920 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:02.407938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:02.412268 (+ 4330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260809 06:20:02.413050 13460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:02 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:02.842630 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:02.830825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:50789)
0809 06:20:02.831871 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:02.831917 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:02.832095 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:02.832612 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:02.832617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:02.833170 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:02.833405 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.833416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.835435 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.835439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.836413 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:02.836425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.837828 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.837832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:02.838028 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.838653 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:02.838670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:02.842280 (+ 3610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":624,"threads_started":1}
W20260809 06:20:02.843343 13462 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:02.831122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:50789 (local address 127.3.137.126:37999)
0809 06:20:02.831359 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:02.831368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:02.832025 (+ 657us) server_negotiation.cc:407] Connection header received
0809 06:20:02.832246 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:02.832256 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:02.832407 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:02.832558 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:02.833574 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.835270 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.836586 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:02.837634 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:02.837811 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:02.842637 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:50789: BlockingRecv error: recv got EOF from 127.3.137.65:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 06:20:03.069114 13603 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.110s user 0.004s sys 0.011s
W20260809 06:20:03.070638 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:01.917590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:48191 (local address 127.3.137.126:37999)
0809 06:20:01.937431 (+ 19841us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:01.937441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:01.943046 (+ 5605us) server_negotiation.cc:407] Connection header received
0809 06:20:01.948906 (+ 5860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:01.948912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:01.949001 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:01.949148 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:01.951242 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.953179 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.955920 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.956981 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.957207 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.070381 (+1113174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:48191: BlockingRecv error: recv got EOF from 127.3.137.67:48191 (error 108)
Metrics: {"server-negotiator.queue_time_us":19600,"thread_start_us":19880,"threads_started":1}
I20260809 06:20:03.071220 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:01.916873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:48191)
0809 06:20:01.938547 (+ 21674us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:01.938630 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:01.942861 (+ 4231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:01.949468 (+ 6607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:01.949489 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:01.950064 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:01.950955 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.951035 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.953364 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.953371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.955699 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:01.955720 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:01.958361 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:01.958372 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:01.958636 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:01.959328 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:01.959350 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:03.070086 (+1110736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21119,"thread_start_us":26074,"threads_started":1}
W20260809 06:20:03.072167 13600 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:37999 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.137.126:37999: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:03.081669 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:03.072851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:47699)
0809 06:20:03.073032 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:03.073052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:03.073152 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:03.073439 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:03.073441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:03.073777 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:03.073941 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.073948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.074923 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.074926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.076262 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.076286 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.077087 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.077093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.077257 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.077885 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:03.077903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:03.081356 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 06:20:03.081799 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:03.072880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47699 (local address 127.3.137.126:37999)
0809 06:20:03.073108 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:03.073113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:03.073130 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:20:03.073254 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:03.073258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:03.073319 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:03.073399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:03.074059 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.074805 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.076427 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.076842 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.077038 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.081561 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:47699: BlockingRecv error: recv got EOF from 127.3.137.67:47699 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:03.092039 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:03.082378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:33997)
0809 06:20:03.082527 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:03.082543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:03.082622 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:03.082863 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:03.082867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:03.083226 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:03.083388 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.083396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.084434 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.084437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.085383 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.085394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.086119 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.086123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.086308 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.086889 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:03.086911 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:03.091735 (+ 4824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 06:20:03.092370 13600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:20:03.092528 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:03.082482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:33997 (local address 127.3.137.126:37999)
0809 06:20:03.082632 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:03.082636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:03.082651 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:20:03.082693 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:03.082697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:03.082750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:03.082826 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:03.083520 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.084310 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.085499 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.085973 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.086065 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.091995 (+ 5930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:33997: BlockingRecv error: recv got EOF from 127.3.137.67:33997 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:03.423478 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:03.413589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:48065)
0809 06:20:03.414227 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:03.414244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:03.414391 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:03.414665 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:03.414668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:03.415190 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:03.415349 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.415359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.416323 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.416327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.417301 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.417312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.418026 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.418029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.418144 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.418716 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:03.418733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:03.423156 (+ 4423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":397,"threads_started":1}
W20260809 06:20:03.423522 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:03.413676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:48065 (local address 127.3.137.126:37999)
0809 06:20:03.413869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:03.413874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:03.414329 (+ 455us) server_negotiation.cc:407] Connection header received
0809 06:20:03.414474 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:03.414477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:03.414542 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:03.414622 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:03.415493 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.416194 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.417448 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.417857 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.417955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.423342 (+ 5387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:48065: BlockingRecv error: recv got EOF from 127.3.137.66:48065 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:03 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:03.854437 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:03.843635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:42755)
0809 06:20:03.844181 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:03.844202 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:03.844374 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:03.844710 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:03.844714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:03.845157 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:03.845351 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.845360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.846517 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.846523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.848160 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:03.848179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.849227 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.849233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:03.849462 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.850153 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:03.850171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:03.854088 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":326,"threads_started":1}
W20260809 06:20:03.854552 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:03.843727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:42755 (local address 127.3.137.126:37999)
0809 06:20:03.843978 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:03.843987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:03.844294 (+ 307us) server_negotiation.cc:407] Connection header received
0809 06:20:03.844509 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:03.844513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:03.844575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:03.844676 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:03.845520 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.846340 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.848326 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:03.849025 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:03.849159 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:03.854296 (+ 5137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:42755: BlockingRecv error: recv got EOF from 127.3.137.65:42755 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:04.102031 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:04.093035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:48803)
0809 06:20:04.093640 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:04.093656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:04.093768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:04.094129 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:04.094132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:04.094561 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:04.094721 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.094731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.095758 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.095762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.096718 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.096730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.097481 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.097485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.097608 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.098169 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:04.098219 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:04.101770 (+ 3551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":358,"threads_started":1}
W20260809 06:20:04.102308 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:04.093115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:48803 (local address 127.3.137.126:37999)
0809 06:20:04.093315 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:04.093321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:04.093745 (+ 424us) server_negotiation.cc:407] Connection header received
0809 06:20:04.093878 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:04.093883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:04.093978 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:04.094087 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:04.094856 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.095626 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.096872 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.097303 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.097404 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.102072 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:48803: BlockingRecv error: recv got EOF from 127.3.137.67:48803 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:04.433301 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:04.424432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:50321)
0809 06:20:04.425015 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:04.425032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:04.425149 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:04.425494 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:04.425497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:04.425919 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:04.426114 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.426124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.427228 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.427232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.428165 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.428177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.428823 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.428827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.428962 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.429537 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:04.429554 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:04.433039 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":341,"threads_started":1}
W20260809 06:20:04.433483 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:04.424556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:50321 (local address 127.3.137.126:37999)
0809 06:20:04.424725 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:04.424729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:04.425139 (+ 410us) server_negotiation.cc:407] Connection header received
0809 06:20:04.425282 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:04.425292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:04.425357 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:04.425449 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:04.426257 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.427083 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.428299 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.428691 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.428785 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.433243 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:50321: BlockingRecv error: recv got EOF from 127.3.137.66:50321 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:04 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:04.867844 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:04.855536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:38743)
0809 06:20:04.856407 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:04.856457 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:04.856661 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:04.857264 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:04.857272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:04.857849 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:04.858154 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.858169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.859428 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.859437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.861144 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:04.861182 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.862262 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.862275 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:04.862511 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.863368 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:04.863396 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:04.867468 (+ 4072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":585,"threads_started":1}
W20260809 06:20:04.867916 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:04.855644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:38743 (local address 127.3.137.126:37999)
0809 06:20:04.856131 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:04.856135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:04.856601 (+ 466us) server_negotiation.cc:407] Connection header received
0809 06:20:04.856795 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:04.856799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:04.856863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:04.857001 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:04.858362 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.859258 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.861318 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:04.862079 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:04.862249 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:04.867672 (+ 5423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:38743: BlockingRecv error: recv got EOF from 127.3.137.65:38743 (error 108)
Metrics: {"server-negotiator.queue_time_us":315}
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:05.112672 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:05.103277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:59285)
0809 06:20:05.103832 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:05.103848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:05.103967 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:05.104261 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:05.104264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:05.104675 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:05.104867 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.104876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.106048 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.106055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.107192 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.107210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.108010 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.108016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.108209 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.108805 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:05.108822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:05.112413 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":337,"threads_started":1}
W20260809 06:20:05.112879 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:05.103379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:59285 (local address 127.3.137.126:37999)
0809 06:20:05.103546 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:05.103550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:05.103925 (+ 375us) server_negotiation.cc:407] Connection header received
0809 06:20:05.104064 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:05.104067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:05.104131 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:05.104217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:05.105042 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.105903 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.107344 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.107848 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.107997 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.112604 (+ 4607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:59285: BlockingRecv error: recv got EOF from 127.3.137.67:59285 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:05.443521 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:05.434326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:43009)
0809 06:20:05.434881 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:05.434898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:05.435066 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:05.435366 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:05.435369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:05.435724 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:05.435895 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.435903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.437261 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.437266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.438305 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.438318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.438954 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.439004 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.439158 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.439751 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:05.439769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:05.443264 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":285,"threads_started":1}
W20260809 06:20:05.443763 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:05.434460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:43009 (local address 127.3.137.126:37999)
0809 06:20:05.434657 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:05.434662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:05.434976 (+ 314us) server_negotiation.cc:407] Connection header received
0809 06:20:05.435169 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:05.435173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:05.435234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:05.435321 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:05.436047 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.437106 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.438434 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.438826 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.438921 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.443518 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:43009: BlockingRecv error: recv got EOF from 127.3.137.66:43009 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:05 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:05.882323 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:05.868768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:40425)
0809 06:20:05.869377 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:05.869400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:05.869509 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:05.869798 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:05.869802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:05.870253 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:05.870446 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.870456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.871520 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.871524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.872523 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:05.872536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.873169 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.873173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:05.873304 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.878436 (+ 5132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:05.878461 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:05.882049 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":357,"threads_started":1}
W20260809 06:20:05.882513 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:05.868827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:40425 (local address 127.3.137.126:37999)
0809 06:20:05.869142 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:05.869146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:05.869473 (+ 327us) server_negotiation.cc:407] Connection header received
0809 06:20:05.869602 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:05.869606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:05.869668 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:05.869748 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:05.870574 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.871389 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.872647 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:05.873042 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:05.873131 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:05.882249 (+ 9118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:40425: BlockingRecv error: recv got EOF from 127.3.137.65:40425 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:06.123490 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:06.113737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:58841)
0809 06:20:06.114336 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:06.114362 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:06.114483 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:06.115043 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:06.115048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:06.115550 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:06.115730 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.115740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.116804 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.116808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.117726 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.117737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.118398 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.118401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.118548 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.119206 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:06.119224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:06.123169 (+ 3945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":312,"threads_started":1}
W20260809 06:20:06.123524 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:06.113846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:58841 (local address 127.3.137.126:37999)
0809 06:20:06.114103 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:06.114110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:06.114453 (+ 343us) server_negotiation.cc:407] Connection header received
0809 06:20:06.114590 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:06.114601 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:06.114665 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:06.114768 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:06.115870 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.116687 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.117858 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.118258 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.118367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.123355 (+ 4988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:58841: BlockingRecv error: recv got EOF from 127.3.137.67:58841 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:06.455029 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:06.444507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:35481)
0809 06:20:06.445115 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:06.445131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:06.445285 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:06.445686 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:06.445692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:06.446212 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:06.446457 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.446474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.448049 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.448056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.449617 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.449630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.450330 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.450335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.450451 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.451143 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:06.451160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:06.454618 (+ 3458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":340,"threads_started":1}
W20260809 06:20:06.455122 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:06.444757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:35481 (local address 127.3.137.126:37999)
0809 06:20:06.444983 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:06.444991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:06.445215 (+ 224us) server_negotiation.cc:407] Connection header received
0809 06:20:06.445406 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:06.445411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:06.445493 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:06.445600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:06.446671 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.447860 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.449751 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.450191 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.450283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.454842 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:35481: BlockingRecv error: recv got EOF from 127.3.137.66:35481 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:06 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:06.892926 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:06.883381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:52667)
0809 06:20:06.883969 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:06.883990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:06.884107 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:06.884413 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:06.884417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:06.884860 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:06.885024 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.885033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.886374 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.886379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.887695 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:06.887709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.888351 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.888354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:06.888487 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.889045 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:06.889062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:06.892594 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":322,"threads_started":1}
W20260809 06:20:06.893055 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:06.883446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:52667 (local address 127.3.137.126:37999)
0809 06:20:06.883637 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:06.883641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:06.884075 (+ 434us) server_negotiation.cc:407] Connection header received
0809 06:20:06.884208 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:06.884212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:06.884281 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:06.884368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:06.885170 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.886205 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.887845 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:06.888230 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:06.888327 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:06.892812 (+ 4485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:52667: BlockingRecv error: recv got EOF from 127.3.137.65:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:07.133515 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:07.124722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:36545)
0809 06:20:07.125270 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:07.125312 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:07.125474 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:07.125796 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:07.125799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:07.126166 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:07.126353 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.126363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.127408 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.127412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.128421 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.128432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.129087 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.129091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.129219 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.129806 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:07.129824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:07.133274 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":409,"threads_started":1}
W20260809 06:20:07.133692 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:07.124762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:36545 (local address 127.3.137.126:37999)
0809 06:20:07.125032 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:07.125039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:07.125405 (+ 366us) server_negotiation.cc:407] Connection header received
0809 06:20:07.125594 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:07.125598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:07.125659 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:07.125738 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:07.126483 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.127289 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.128545 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.128939 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.129037 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.133463 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:36545: BlockingRecv error: recv got EOF from 127.3.137.67:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:07.465652 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:07.456016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:54169)
0809 06:20:07.456585 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:07.456606 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:07.456732 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:07.457143 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:07.457149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:07.457771 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:07.457987 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.458002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.459289 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.459295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.460300 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.460314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.460997 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.461001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.461136 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.461795 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:07.461813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:07.465402 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":304,"threads_started":1}
W20260809 06:20:07.465824 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:07.456220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:54169 (local address 127.3.137.126:37999)
0809 06:20:07.456400 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:07.456405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:07.456694 (+ 289us) server_negotiation.cc:407] Connection header received
0809 06:20:07.456851 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:07.456857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:07.456966 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:07.457089 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:07.458128 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.459117 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.460442 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.460858 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.460955 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.465598 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:54169: BlockingRecv error: recv got EOF from 127.3.137.66:54169 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:07 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:07.903834 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:07.894118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:49533)
0809 06:20:07.894694 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:07.894737 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:07.894914 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:07.895349 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:07.895354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:07.895833 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:07.896020 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.896030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.897185 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.897195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.898224 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:07.898238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.899105 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.899112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:07.899323 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.899883 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:07.899900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:07.903434 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":338,"threads_started":1}
W20260809 06:20:07.903926 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:07.894226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:49533 (local address 127.3.137.126:37999)
0809 06:20:07.894450 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:07.894459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:07.894840 (+ 381us) server_negotiation.cc:407] Connection header received
0809 06:20:07.895115 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:07.895125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:07.895193 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:07.895315 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:07.896182 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.897016 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.898383 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:07.898890 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:07.899082 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:07.903650 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:49533: BlockingRecv error: recv got EOF from 127.3.137.65:49533 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:08.143329 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:08.134647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:50899)
0809 06:20:08.135205 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:08.135221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:08.135333 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:08.135598 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:08.135601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:08.136049 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:08.136194 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.136202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.137195 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.137199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.138115 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.138126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.138774 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.138778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.138926 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.139503 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:08.139519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:08.143098 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":315,"threads_started":1}
W20260809 06:20:08.143436 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:08.134681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:50899 (local address 127.3.137.126:37999)
0809 06:20:08.134900 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:08.134909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:08.135313 (+ 404us) server_negotiation.cc:407] Connection header received
0809 06:20:08.135425 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:08.135429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:08.135490 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:08.135570 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:08.136332 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.137078 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.138240 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.138652 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.138747 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.143219 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:50899: BlockingRecv error: recv got EOF from 127.3.137.67:50899 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:08.476208 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:08.466706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:43041)
0809 06:20:08.467324 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:08.467340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:08.467520 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:08.467860 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:08.467864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:08.468299 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:08.468469 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.468479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.469459 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.469465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.470429 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.470440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.471203 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.471209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.471408 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.471994 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:08.472012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:08.475825 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":350,"threads_started":1}
W20260809 06:20:08.476364 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:08.466934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:43041 (local address 127.3.137.126:37999)
0809 06:20:08.467194 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:08.467200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:08.467427 (+ 227us) server_negotiation.cc:407] Connection header received
0809 06:20:08.467648 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:08.467659 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:08.467719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:08.467822 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:08.468618 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.469305 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.470585 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.471049 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.471179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.476070 (+ 4891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:43041: BlockingRecv error: recv got EOF from 127.3.137.66:43041 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:08 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:08.914844 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:08.904877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:54065)
0809 06:20:08.905578 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:08.905599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:08.905760 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:08.906155 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:08.906160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:08.906648 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:08.906898 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.906921 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.907970 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.907976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.909058 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:08.909075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.909782 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.909788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:08.909977 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.910589 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:08.910606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:08.914602 (+ 3996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":320,"threads_started":1}
W20260809 06:20:08.915053 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:08.905060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:54065 (local address 127.3.137.126:37999)
0809 06:20:08.905325 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:08.905334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:08.905690 (+ 356us) server_negotiation.cc:407] Connection header received
0809 06:20:08.905926 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:08.905930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:08.906002 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:08.906105 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:08.907095 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.907809 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.909206 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:08.909621 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:08.909751 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:08.914828 (+ 5077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:54065: BlockingRecv error: recv got EOF from 127.3.137.65:54065 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:09.154366 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:09.144433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:36381)
0809 06:20:09.144920 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:09.144936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:09.145102 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:09.145466 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:09.145473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:09.145976 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:09.146152 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.146162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.147386 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.147392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.148366 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.148378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.149156 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.149162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.149343 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.150002 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:09.150020 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:09.154132 (+ 4112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":296,"threads_started":1}
W20260809 06:20:09.154697 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:09.144514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:36381 (local address 127.3.137.126:37999)
0809 06:20:09.144748 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:09.144755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:09.145031 (+ 276us) server_negotiation.cc:407] Connection header received
0809 06:20:09.145239 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:09.145245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:09.145316 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:09.145419 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:09.146298 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.147230 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.148541 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.148998 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.149138 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.154349 (+ 5211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:36381: BlockingRecv error: recv got EOF from 127.3.137.67:36381 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:09.486917 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:09.477369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:53693)
0809 06:20:09.477989 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:09.478006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:09.478137 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:09.478529 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:09.478532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:09.479091 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:09.479279 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.479289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.480356 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.480359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.481305 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.481316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.482039 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.482043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.482154 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.482719 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:09.482736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:09.486668 (+ 3932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":299,"threads_started":1}
W20260809 06:20:09.487097 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:09.477434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:53693 (local address 127.3.137.126:37999)
0809 06:20:09.477679 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:09.477683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:09.478094 (+ 411us) server_negotiation.cc:407] Connection header received
0809 06:20:09.478300 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:09.478305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:09.478379 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:09.478488 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:09.479441 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.480222 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.481448 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.481904 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.482001 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.486860 (+ 4859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:53693: BlockingRecv error: recv got EOF from 127.3.137.66:53693 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:09 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:09.925048 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:09.915992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:38845)
0809 06:20:09.916534 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:09.916554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:09.916709 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:09.917061 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:09.917066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:09.917508 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:09.917701 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.917710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.918739 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.918746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.919747 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:09.919759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.920577 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.920583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:09.920757 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.921339 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:09.921356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:09.924800 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
W20260809 06:20:09.925273 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:09.916078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:38845 (local address 127.3.137.126:37999)
0809 06:20:09.916318 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:09.916327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:09.916644 (+ 317us) server_negotiation.cc:407] Connection header received
0809 06:20:09.916839 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:09.916844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:09.916920 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:09.917025 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:09.917851 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.918597 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.919898 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:09.920370 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:09.920531 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:09.925006 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:38845: BlockingRecv error: recv got EOF from 127.3.137.65:38845 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:10.166694 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:10.155641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:37479)
0809 06:20:10.156391 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:10.156412 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:10.156555 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:10.157038 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:10.157045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:10.157754 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:10.158030 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.158045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.159805 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.159811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.161156 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.161170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.162196 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.162202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.162394 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.163099 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:10.163116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:10.166457 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":324,"threads_started":1}
W20260809 06:20:10.166944 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:10.156441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:37479 (local address 127.3.137.126:37999)
0809 06:20:10.156674 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:10.156681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:10.156699 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:20:10.156756 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:10.156761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:10.156836 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:10.156958 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:10.158224 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.159626 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.161301 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.162022 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.162173 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.166698 (+ 4525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:37479: BlockingRecv error: recv got EOF from 127.3.137.67:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:10.496897 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:10.488045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:59641)
0809 06:20:10.488637 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:10.488663 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:10.488777 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:10.489071 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:10.489074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:10.489480 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:10.489649 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.489658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.490838 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.490842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.491797 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.491808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.492480 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.492484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.492598 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.493263 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:10.493281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:10.496579 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":291,"threads_started":1}
W20260809 06:20:10.496999 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:10.488144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:59641 (local address 127.3.137.126:37999)
0809 06:20:10.488420 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:10.488426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:10.488726 (+ 300us) server_negotiation.cc:407] Connection header received
0809 06:20:10.488880 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:10.488884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:10.488947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:10.489027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:10.489775 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.490718 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.491918 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.492325 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.492420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.496758 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:59641: BlockingRecv error: recv got EOF from 127.3.137.66:59641 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:10 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:10.946874 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:10.926092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:33967)
0809 06:20:10.926751 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:10.926772 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:10.926939 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:10.927455 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:10.927460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:10.927994 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:10.928162 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.928172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.929251 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.929256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.930236 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:10.930248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.931231 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.931238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:10.931430 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.942859 (+ 11429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:10.942880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:10.946634 (+ 3754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":404,"threads_started":1}
W20260809 06:20:10.947230 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:10.926171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:33967 (local address 127.3.137.126:37999)
0809 06:20:10.926405 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:10.926412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:10.926869 (+ 457us) server_negotiation.cc:407] Connection header received
0809 06:20:10.927168 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:10.927175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:10.927273 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:10.927390 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:10.928330 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.929099 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.930387 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:10.931055 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:10.931185 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:10.946925 (+ 15740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:33967: BlockingRecv error: recv got EOF from 127.3.137.65:33967 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:11.177516 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:11.167762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:53021)
0809 06:20:11.168348 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:11.168364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:11.168533 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:11.168886 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:11.168891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:11.169328 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:11.169556 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.169566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.170779 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.170784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.171797 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.171810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.172611 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.172618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.172791 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.173350 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:11.173367 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:11.177272 (+ 3905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":335,"threads_started":1}
W20260809 06:20:11.177727 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:11.167977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:53021 (local address 127.3.137.126:37999)
0809 06:20:11.168179 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:11.168185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:11.168450 (+ 265us) server_negotiation.cc:407] Connection header received
0809 06:20:11.168668 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:11.168673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:11.168745 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:11.168849 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:11.169697 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.170623 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.171945 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.172390 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.172524 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.177475 (+ 4951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:53021: BlockingRecv error: recv got EOF from 127.3.137.67:53021 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:11.507781 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:11.498036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:45519)
0809 06:20:11.498597 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:11.498614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:11.498772 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:11.499168 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:11.499174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:11.499631 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:11.499780 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.499791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.500789 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.500796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.501818 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.501830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.502547 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.502552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.502740 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.503375 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:11.503393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:11.507432 (+ 4039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":303,"threads_started":1}
W20260809 06:20:11.507882 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:11.498101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:45519 (local address 127.3.137.126:37999)
0809 06:20:11.498304 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:11.498308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:11.498710 (+ 402us) server_negotiation.cc:407] Connection header received
0809 06:20:11.498890 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:11.498894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:11.498986 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:11.499113 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:11.499935 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.500649 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.501971 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.502387 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.502517 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.507628 (+ 5111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:45519: BlockingRecv error: recv got EOF from 127.3.137.66:45519 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:11 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:11.957392 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:11.947970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:50993)
0809 06:20:11.948559 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:11.948580 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:11.948697 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:11.949047 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:11.949050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:11.949437 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:11.949609 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.949618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.950728 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.950733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.951856 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:11.951869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.952564 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.952568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:11.952683 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.953266 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:11.953283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:11.957013 (+ 3730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":353,"threads_started":1}
W20260809 06:20:11.957544 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:11.948017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:50993 (local address 127.3.137.126:37999)
0809 06:20:11.948210 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:11.948220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:11.948669 (+ 449us) server_negotiation.cc:407] Connection header received
0809 06:20:11.948842 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:11.948847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:11.948913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:11.949004 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:11.949785 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.950584 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.952027 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:11.952437 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:11.952531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:11.957227 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:50993: BlockingRecv error: recv got EOF from 127.3.137.65:50993 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:12.187501 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:12.178504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:59709)
0809 06:20:12.179093 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:12.179110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:12.179222 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:12.179508 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:12.179511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:12.179944 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:12.180099 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.180108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.181084 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.181087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.182113 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.182125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.182883 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.182888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.183124 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.183806 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:12.183824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:12.187261 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":316,"threads_started":1}
W20260809 06:20:12.187640 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:12.178563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:59709 (local address 127.3.137.126:37999)
0809 06:20:12.178748 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:12.178752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:12.179182 (+ 430us) server_negotiation.cc:407] Connection header received
0809 06:20:12.179333 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:12.179337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:12.179399 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:12.179476 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:12.180221 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.180968 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.182243 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.182689 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.182842 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.187438 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:59709: BlockingRecv error: recv got EOF from 127.3.137.67:59709 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:12.518276 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:12.508639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:58701)
0809 06:20:12.509154 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:12.509171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:12.509309 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:12.509681 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:12.509687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:12.510105 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:12.510271 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.510281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.511339 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.511344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.512344 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.512355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.513106 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.513112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.513290 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.514066 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:12.514104 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:12.517928 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
W20260809 06:20:12.518363 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:12.508735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:58701 (local address 127.3.137.126:37999)
0809 06:20:12.509019 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:12.509025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:12.509272 (+ 247us) server_negotiation.cc:407] Connection header received
0809 06:20:12.509443 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:12.509448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:12.509529 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:12.509639 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:12.510414 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.511204 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.512501 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.512952 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.513088 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.518106 (+ 5018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:58701: BlockingRecv error: recv got EOF from 127.3.137.66:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:12 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:12.967473 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:12.958502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:40409)
0809 06:20:12.959049 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:12.959071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:12.959186 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:12.959489 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:12.959492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:12.959921 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:12.960108 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.960116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.961077 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.961080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.962015 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:12.962027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.962716 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.962720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:12.962846 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.963441 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:12.963464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:12.967093 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
W20260809 06:20:12.967630 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:12.958679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:40409 (local address 127.3.137.126:37999)
0809 06:20:12.958923 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:12.958930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:12.959146 (+ 216us) server_negotiation.cc:407] Connection header received
0809 06:20:12.959292 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:12.959296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:12.959359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:12.959440 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:12.960230 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.960967 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.962147 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:12.962590 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:12.962683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:12.967325 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:40409: BlockingRecv error: recv got EOF from 127.3.137.65:40409 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:13.197256 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:13.188538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:34881)
0809 06:20:13.189080 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:13.189096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:13.189212 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:13.189518 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:13.189521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:13.189885 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:13.190068 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.190077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.191175 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.191179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.192188 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.192200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.192884 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.192888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.193014 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.193587 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:13.193604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:13.197000 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
W20260809 06:20:13.197449 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:13.188620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:34881 (local address 127.3.137.126:37999)
0809 06:20:13.188800 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:13.188804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:13.189172 (+ 368us) server_negotiation.cc:407] Connection header received
0809 06:20:13.189316 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:13.189320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:13.189384 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:13.189468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:13.190196 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.191055 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.192316 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.192749 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.192852 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.197192 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:34881: BlockingRecv error: recv got EOF from 127.3.137.67:34881 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:13.528676 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:13.519340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:49473)
0809 06:20:13.519933 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:13.519949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:13.520101 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:13.520436 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:13.520444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:13.520891 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:13.521076 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.521085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.522108 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.522116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.523118 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.523131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.523838 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.523844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.524036 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.524620 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:13.524637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:13.528456 (+ 3819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":322,"threads_started":1}
W20260809 06:20:13.528929 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:13.519463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:49473 (local address 127.3.137.126:37999)
0809 06:20:13.519702 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:13.519709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:13.520040 (+ 331us) server_negotiation.cc:407] Connection header received
0809 06:20:13.520221 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:13.520225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:13.520286 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:13.520392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:13.521227 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.521975 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.523256 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.523688 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.523823 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.528658 (+ 4835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:49473: BlockingRecv error: recv got EOF from 127.3.137.66:49473 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:13 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:13.978487 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:13.968654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:48063)
0809 06:20:13.969291 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:13.969312 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:13.969462 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:13.969816 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:13.969819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:13.970253 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:13.970467 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.970483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.972031 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.972039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.973123 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:13.973138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.973865 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.973869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:13.974013 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.974627 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:13.974663 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:13.978242 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":369,"threads_started":1}
W20260809 06:20:13.978847 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:13.968807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48063 (local address 127.3.137.126:37999)
0809 06:20:13.968984 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:13.968989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:13.969389 (+ 400us) server_negotiation.cc:407] Connection header received
0809 06:20:13.969607 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:13.969610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:13.969675 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:13.969760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:13.970691 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.971854 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.973283 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:13.973694 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:13.973809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:13.978446 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:48063: BlockingRecv error: recv got EOF from 127.3.137.65:48063 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:20:14.207538 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:14.198234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:42955 (local address 127.3.137.126:37999)
0809 06:20:14.198481 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:14.198488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:14.198699 (+ 211us) server_negotiation.cc:407] Connection header received
0809 06:20:14.198863 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:14.198866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:14.198928 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:14.199027 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:14.199832 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.200566 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.201907 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.202302 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.202392 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.207329 (+ 4937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:42955: BlockingRecv error: recv got EOF from 127.3.137.67:42955 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:20:14.207581 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:14.198139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:42955)
0809 06:20:14.198603 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:14.198619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:14.198738 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:14.199083 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:14.199086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:14.199509 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:14.199705 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.199713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.200708 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.200723 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.201774 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.201788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.202442 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.202446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.202589 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.203253 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:14.203273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:14.207163 (+ 3890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:14.538492 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:14.529552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:37053)
0809 06:20:14.530039 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:14.530055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:14.530185 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:14.530470 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:14.530473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:14.530862 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:14.531053 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.531063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.532060 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.532063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.532994 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.533006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.533861 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.533865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.533998 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.534556 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:14.534573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:14.538165 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
W20260809 06:20:14.538687 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:14.529622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:37053 (local address 127.3.137.126:37999)
0809 06:20:14.529882 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:14.529889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:14.530151 (+ 262us) server_negotiation.cc:407] Connection header received
0809 06:20:14.530282 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:14.530285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:14.530347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:14.530427 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:14.531175 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.531933 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.533138 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.533692 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.533788 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.538394 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:37053: BlockingRecv error: recv got EOF from 127.3.137.66:37053 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:14 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:14.988408 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:14.979443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:56077)
0809 06:20:14.979956 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:14.979984 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:14.980142 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:14.980438 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:14.980441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:14.980880 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:14.981031 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.981040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.981995 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.981999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.982915 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:14.982927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.983675 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.983679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:14.983808 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.984374 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:14.984391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:14.988082 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
W20260809 06:20:14.988538 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:14.979543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:56077 (local address 127.3.137.126:37999)
0809 06:20:14.979794 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:14.979801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:14.980089 (+ 288us) server_negotiation.cc:407] Connection header received
0809 06:20:14.980250 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:14.980254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:14.980314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:14.980392 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:14.981156 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.981864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.983104 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:14.983544 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:14.983641 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:14.988282 (+ 4641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:56077: BlockingRecv error: recv got EOF from 127.3.137.65:56077 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:15.218349 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:15.208691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:47659)
0809 06:20:15.209273 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:15.209289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:15.209466 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:15.209832 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:15.209837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:15.210322 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:15.210507 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.210516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.211651 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.211657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.212900 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.212914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.213695 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.213701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.213884 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:15.214527 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:15.214545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:15.217987 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":322,"threads_started":1}
W20260809 06:20:15.218464 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:15.208747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47659 (local address 127.3.137.126:37999)
0809 06:20:15.208996 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:15.209004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:15.209380 (+ 376us) server_negotiation.cc:407] Connection header received
0809 06:20:15.209604 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:15.209609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:15.209680 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:15.209781 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:15.210657 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.211499 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.213047 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.213509 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.213646 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:15.218195 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:47659: BlockingRecv error: recv got EOF from 127.3.137.67:47659 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:15.548856 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:15.539471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:39405)
0809 06:20:15.539995 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:15.540010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:15.540165 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:15.540502 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:15.540507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:15.540970 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:15.541119 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.541128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.542147 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.542158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.543124 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.543135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.544201 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.544207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.544413 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:15.545062 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:15.545081 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:15.548620 (+ 3539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
W20260809 06:20:15.549043 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:15.539568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39405 (local address 127.3.137.126:37999)
0809 06:20:15.539777 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:15.539784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:15.540098 (+ 314us) server_negotiation.cc:407] Connection header received
0809 06:20:15.540295 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:15.540300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:15.540363 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:15.540474 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:15.541269 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.542004 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.543361 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.544049 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.544190 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:15.548814 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:39405: BlockingRecv error: recv got EOF from 127.3.137.66:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:15 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.65@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:16.004343 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:15.989734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.65:54211)
0809 06:20:15.990241 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:15.990262 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:15.990440 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:15.990780 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:15.990804 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:15.991262 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:15.991469 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.991479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.992547 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.992553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.993628 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:15.993648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.994420 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.994425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:15.994611 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.000502 (+ 5891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:16.000521 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:16.003954 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
W20260809 06:20:16.004402 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:15.989826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:54211 (local address 127.3.137.126:37999)
0809 06:20:15.990084 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:15.990090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:15.990357 (+ 267us) server_negotiation.cc:407] Connection header received
0809 06:20:15.990570 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:15.990580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:15.990643 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:15.990743 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:15.991618 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.992389 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.993768 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:15.994270 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:15.994409 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.004162 (+ 9753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.65:54211: BlockingRecv error: recv got EOF from 127.3.137.65:54211 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 06:20:16 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:16 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.67@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:16.228055 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:16.219571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.67:33197)
0809 06:20:16.220084 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:16.220118 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:16.220269 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:16.220642 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:16.220646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:16.221065 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:16.221214 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:16.221223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.222245 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.222248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:16.223193 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:16.223204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.223875 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.223878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:16.224003 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.224585 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:16.224603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:16.227823 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":286,"threads_started":1}
W20260809 06:20:16.228379 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:16.219619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:33197 (local address 127.3.137.126:37999)
0809 06:20:16.219931 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:16.219939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:16.220223 (+ 284us) server_negotiation.cc:407] Connection header received
0809 06:20:16.220443 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:16.220449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:16.220523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:16.220598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:16.221362 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.222128 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.223337 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.223750 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.223853 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.228064 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.67:33197: BlockingRecv error: recv got EOF from 127.3.137.67:33197 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Aug 09 06:20:16 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:20:16 dist-test-slave-2w3w krb5kdc[12540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.137.66@KRBTEST.COM for oryx/127.3.137.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:20:16.559505 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:16.550246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:37999 (local address 127.3.137.66:50755)
0809 06:20:16.550844 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:16.550860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:16.551012 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:16.551424 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:16.551430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:16.551906 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:16.552099 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:16.552109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.553123 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.553129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:16.554068 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:16.554079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.554845 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.554851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:16.555074 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.555625 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:16.555642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:16.559133 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.137.126:37999: Server oryx/127.3.137.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":304,"threads_started":1}
W20260809 06:20:16.559618 13604 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:16.550305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:50755 (local address 127.3.137.126:37999)
0809 06:20:16.550494 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:16.550501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:16.550934 (+ 433us) server_negotiation.cc:407] Connection header received
0809 06:20:16.551158 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:16.551164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:16.551266 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:16.551372 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:16.552247 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.552979 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.554207 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.554688 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.554833 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.559327 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.137.66:50755: BlockingRecv error: recv got EOF from 127.3.137.66:50755 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:20:16.918381 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13165
I20260809 06:20:16.931030 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:16.919560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.3.137.65:46647)
0809 06:20:16.920272 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:16.920281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:16.920307 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:20:16.920371 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:16.920377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:16.920626 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:16.920799 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:16.922261 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.923393 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.924591 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.925122 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.925224 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.927818 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:16.927826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:16.927839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:16.927890 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:16.930476 (+ 2586us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:20:16.930762 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57884: Request ticket server kudu/127.3.137.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":392,"threads_started":1}
W20260809 06:20:16.931226 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13165: Not authorized: Client connection negotiation failed: client connection to 127.3.137.65:46647: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:20:16.931429 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13165
I20260809 06:20:16.948596 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13324
I20260809 06:20:16.963558 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:16.949821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.3.137.66:42901)
0809 06:20:16.950573 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:16.950637 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:16.950681 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:20:16.950811 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:16.950819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:16.951158 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:16.951466 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:16.953266 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.954414 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.955797 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.956674 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.956900 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.960075 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:16.960085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:16.960099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:16.960169 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:16.962800 (+ 2631us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:20:16.963199 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35310: Request ticket server kudu/127.3.137.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":423,"threads_started":1}
W20260809 06:20:16.963562 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13324: Not authorized: Client connection negotiation failed: client connection to 127.3.137.66:42901: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 06:20:16.963671 13651 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35310: Request ticket server kudu/127.3.137.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:20:16.963709 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13324
I20260809 06:20:16.982694 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13464
I20260809 06:20:16.996716 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:16.983961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.3.137.67:33593)
0809 06:20:16.984569 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:16.984577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:16.984603 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:20:16.984675 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:16.984681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:16.984921 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:16.985157 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:16.986690 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.987801 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.989024 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:16.989593 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:16.989744 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:16.992609 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:16.992617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:16.992633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:16.992677 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:16.995962 (+ 3285us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:20:16.996359 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58796: Request ticket server kudu/127.3.137.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
W20260809 06:20:16.996838 13652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58796: Request ticket server kudu/127.3.137.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 06:20:16.996802 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13464: Not authorized: Client connection negotiation failed: client connection to 127.3.137.67:33593: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.137.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:20:16.996995 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13464
I20260809 06:20:17.019591 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13093
I20260809 06:20:17.021157 13158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:17.155447 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13093
2026-08-09T06:20:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21643 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T06:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:20:17Z Disabled control of system clock
I20260809 06:20:17.239337 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:32955
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 06:20:17.571120 13659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:17.571404 13659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:17.571550 13659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:17.578667 13659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:20:17.578766 13659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:17.578857 13659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:17.578974 13659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:20:17.579054 13659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:20:17.588068 13659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:32955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:17.589447 13659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:17.591315 13659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:17.603181 13669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:17.603298 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:17.603268 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:17.604116 13659 server_base.cc:1034] running on GCE node
I20260809 06:20:17.604851 13659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:17.606184 13659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:17.607530 13659 hybrid_clock.cc:648] HybridClock initialized: now 1786256417607485 us; error 51 us; skew 500 ppm
I20260809 06:20:17.607893 13659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:17.610038 13659 webserver.cc:460] Webserver started at http://127.3.137.126:45899/ using document root <none> and password file <none>
I20260809 06:20:17.610707 13659 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:17.610836 13659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:17.611217 13659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:17.614033 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "6b6f5cf4c5e343baa8074a392bda12b4"
format_stamp: "Formatted at 2026-08-09 06:20:17 on dist-test-slave-2w3w"
server_key: "079fdcdedf34fa882dd03fd9a3d144a0"
server_key_iv: "c278e52551587e7b9d58b4f797d36632"
server_key_version: "encryptionkey@0"
I20260809 06:20:17.614832 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "6b6f5cf4c5e343baa8074a392bda12b4"
format_stamp: "Formatted at 2026-08-09 06:20:17 on dist-test-slave-2w3w"
server_key: "079fdcdedf34fa882dd03fd9a3d144a0"
server_key_iv: "c278e52551587e7b9d58b4f797d36632"
server_key_version: "encryptionkey@0"
I20260809 06:20:17.619755 13659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 06:20:17.623090 13675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:17.624303 13659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:20:17.624490 13659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "6b6f5cf4c5e343baa8074a392bda12b4"
format_stamp: "Formatted at 2026-08-09 06:20:17 on dist-test-slave-2w3w"
server_key: "079fdcdedf34fa882dd03fd9a3d144a0"
server_key_iv: "c278e52551587e7b9d58b4f797d36632"
server_key_version: "encryptionkey@0"
I20260809 06:20:17.624706 13659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:17.642195 13659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:17.643043 13659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:17.643326 13659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:17.661549 13659 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:32955
I20260809 06:20:17.661571 13728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:32955 every 8 connection(s)
I20260809 06:20:17.662858 13659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:20:17.666405 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:17.671416 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13659
I20260809 06:20:17.671603 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:20:17.672513 3621 external_mini_cluster.cc:1428] Setting key 2db5f6f4f51ed0a207fa15f389fb6e8a
I20260809 06:20:17.677618 13729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4: Bootstrap starting.
I20260809 06:20:17.681025 13729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:17.682263 13729 log.cc:824] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:17.685560 13729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4: No bootstrap required, opened a new log
I20260809 06:20:17.686655 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:17.674746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.3.137.126:32955)
0809 06:20:17.675830 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:17.675841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:17.675880 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:20:17.677499 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:17.677528 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:17.677992 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:17.678296 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:17.679050 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:17.680300 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:17.682220 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:17.683163 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:17.683366 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:17.684494 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:17.684501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:17.684513 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:17.685027 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:17.685099 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:17.685118 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:17.685196 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:17.685450 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 06:20:17.685776 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":450,"threads_started":1}
I20260809 06:20:17.692900 13729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } }
I20260809 06:20:17.693250 13729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:17.693352 13729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6f5cf4c5e343baa8074a392bda12b4, State: Initialized, Role: FOLLOWER
I20260809 06:20:17.694016 13729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } }
I20260809 06:20:17.694253 13729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:20:17.694404 13729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:20:17.694584 13729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:17.696790 13729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } }
I20260809 06:20:17.697242 13729 leader_election.cc:302] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b6f5cf4c5e343baa8074a392bda12b4; no voters:
I20260809 06:20:17.697789 13729 leader_election.cc:288] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:20:17.697937 13734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:20:17.698613 13734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6f5cf4c5e343baa8074a392bda12b4, State: Running, Role: LEADER
I20260809 06:20:17.699257 13734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } }
I20260809 06:20:17.699767 13729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:20:17.702387 13736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b6f5cf4c5e343baa8074a392bda12b4. Latest consensus state: current_term: 1 leader_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } } }
I20260809 06:20:17.702355 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6f5cf4c5e343baa8074a392bda12b4" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 32955 } } }
I20260809 06:20:17.702831 13736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:17.702868 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:17.704807 13742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:20:17.710158 13742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:20:17.718722 13742 catalog_manager.cc:1347] Generated new cluster ID: 08f9ec8fdbfb4f70b9f56105a5288bb9
I20260809 06:20:17.718833 13742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:20:17.749903 13742 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:20:17.751284 13742 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:20:17.763729 13742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b6f5cf4c5e343baa8074a392bda12b4: Generated new TSK 0
I20260809 06:20:17.764567 13742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:20:17.774544 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--builtin_ntp_servers=127.3.137.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 06:20:18.119776 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:18.120039 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:18.120198 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:18.127288 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:18.127424 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:18.127591 13754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:20:18.137810 13754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:18.139196 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:18.141032 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:18.154667 13762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:18.154773 13760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:18.154752 13759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:18.155362 13754 server_base.cc:1034] running on GCE node
I20260809 06:20:18.156021 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:18.156916 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:18.158145 13754 hybrid_clock.cc:648] HybridClock initialized: now 1786256418158099 us; error 56 us; skew 500 ppm
I20260809 06:20:18.158447 13754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:18.160697 13754 webserver.cc:460] Webserver started at http://127.3.137.65:35237/ using document root <none> and password file <none>
I20260809 06:20:18.161387 13754 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:18.161517 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:18.161836 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:18.164634 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "3f8e44744a4a45e89d7e4216a41e6640"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "f11fb752530d3069352458b92cb51dcf"
server_key_iv: "0a2a299018e055338dc31d3ec9ec291c"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.165421 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "3f8e44744a4a45e89d7e4216a41e6640"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "f11fb752530d3069352458b92cb51dcf"
server_key_iv: "0a2a299018e055338dc31d3ec9ec291c"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.170326 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:20:18.173666 13768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.174870 13754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 06:20:18.175071 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "3f8e44744a4a45e89d7e4216a41e6640"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "f11fb752530d3069352458b92cb51dcf"
server_key_iv: "0a2a299018e055338dc31d3ec9ec291c"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.175282 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:18.188553 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:18.189316 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:18.189610 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:18.190495 13754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:18.191885 13754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:18.192019 13754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.192138 13754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:18.192224 13754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.225806 13754 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:35105
I20260809 06:20:18.225860 13887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:35105 every 8 connection(s)
I20260809 06:20:18.227139 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:20:18.233369 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13754
I20260809 06:20:18.233563 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:20:18.234205 3621 external_mini_cluster.cc:1428] Setting key db359d7879271a431f0e7293069f37e5
I20260809 06:20:18.236769 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--builtin_ntp_servers=127.3.137.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 06:20:18.239382 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:18.229646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:40935 (local address 127.3.137.126:32955)
0809 06:20:18.230099 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:18.230104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:18.231219 (+ 1115us) server_negotiation.cc:407] Connection header received
0809 06:20:18.232926 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:18.232932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:18.233024 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:18.233138 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:18.234205 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.235118 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.236635 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.237118 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.237244 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:18.238526 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:18.238532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:18.238537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:18.238562 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:18.238601 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:18.238613 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:18.238704 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:18.238918 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 06:20:18.239041 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":257,"threads_started":1}
I20260809 06:20:18.240262 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:18.230138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:32955 (local address 127.3.137.65:40935)
0809 06:20:18.231069 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:18.231114 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:18.232673 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:18.233361 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:18.233369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:18.233448 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:18.234045 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:18.234061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.235272 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.235276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:18.236485 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:18.236497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.237338 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.237345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:18.237561 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:18.238236 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:18.238239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:18.238282 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:18.238350 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:18.238703 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:18.238716 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:18.238726 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 06:20:18.238907 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 06:20:18.239286 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":279,"threads_started":1}
I20260809 06:20:18.241773 13888 heartbeater.cc:347] Connected to a master server at 127.3.137.126:32955
I20260809 06:20:18.242048 13888 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:18.242733 13888 heartbeater.cc:511] Master 127.3.137.126:32955 requested a full tablet report, sending...
I20260809 06:20:18.244984 13694 ts_manager.cc:194] Registered new tserver with Master: 3f8e44744a4a45e89d7e4216a41e6640 (127.3.137.65:35105)
I20260809 06:20:18.247172 13694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.65:40935
W20260809 06:20:18.561275 13892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:18.561523 13892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:18.561672 13892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:18.568646 13892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:18.568734 13892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:18.568912 13892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:20:18.577593 13892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:18.578765 13892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:18.580485 13892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:18.593304 13900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:18.593379 13898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:18.593431 13897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:18.599495 13892 server_base.cc:1034] running on GCE node
I20260809 06:20:18.600181 13892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:18.601150 13892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:18.602371 13892 hybrid_clock.cc:648] HybridClock initialized: now 1786256418602333 us; error 46 us; skew 500 ppm
I20260809 06:20:18.602667 13892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:18.605001 13892 webserver.cc:460] Webserver started at http://127.3.137.66:35119/ using document root <none> and password file <none>
I20260809 06:20:18.605916 13892 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:18.606042 13892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:18.606317 13892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:18.609072 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "1f72db17759247f1bb122627e07d7427"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "e612042f366668467e9d5ce69b9f45ed"
server_key_iv: "c9efe955ab9001ecdfe28dac7ba0cd7d"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.609788 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "1f72db17759247f1bb122627e07d7427"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "e612042f366668467e9d5ce69b9f45ed"
server_key_iv: "c9efe955ab9001ecdfe28dac7ba0cd7d"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.614672 13892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:20:18.617996 13906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.619304 13892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:20:18.619467 13892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "1f72db17759247f1bb122627e07d7427"
format_stamp: "Formatted at 2026-08-09 06:20:18 on dist-test-slave-2w3w"
server_key: "e612042f366668467e9d5ce69b9f45ed"
server_key_iv: "c9efe955ab9001ecdfe28dac7ba0cd7d"
server_key_version: "encryptionkey@0"
I20260809 06:20:18.619678 13892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:18.634183 13892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:18.634953 13892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:18.635298 13892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:18.636188 13892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:18.637498 13892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:18.637614 13892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.637737 13892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:18.637828 13892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:18.671698 13892 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:34431
I20260809 06:20:18.671744 14022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:34431 every 8 connection(s)
I20260809 06:20:18.673218 13892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:20:18.683254 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 13892
I20260809 06:20:18.683422 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:20:18.683916 3621 external_mini_cluster.cc:1428] Setting key cc382e051c4c426c54b776ccb1b56fc7
I20260809 06:20:18.685822 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:18.675883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:46221 (local address 127.3.137.126:32955)
0809 06:20:18.676089 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:18.676096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:18.677506 (+ 1410us) server_negotiation.cc:407] Connection header received
0809 06:20:18.679417 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:18.679423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:18.679509 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:18.679618 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:18.680644 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.681414 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.683009 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.683452 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.683591 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:18.684999 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:18.685004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:18.685007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:18.685023 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:18.685054 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:18.685066 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:18.685137 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:18.685421 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 06:20:18.685498 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:20:18.687072 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:18.676345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:32955 (local address 127.3.137.66:46221)
0809 06:20:18.677349 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:18.677390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:18.679112 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:18.679855 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:18.679863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:18.679941 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:18.680477 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:18.680494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.681556 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.681561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:18.682796 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:18.682808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:18.683688 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:18.683696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:18.683937 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:18.684686 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:18.684692 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:18.684745 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:18.684836 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:18.685163 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:18.685194 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:18.685202 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:20:18.685392 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 06:20:18.685771 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":352,"threads_started":1}
I20260809 06:20:18.687229 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--builtin_ntp_servers=127.3.137.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 06:20:18.688527 14023 heartbeater.cc:347] Connected to a master server at 127.3.137.126:32955
I20260809 06:20:18.688815 14023 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:18.689548 14023 heartbeater.cc:511] Master 127.3.137.126:32955 requested a full tablet report, sending...
I20260809 06:20:18.690913 13694 ts_manager.cc:194] Registered new tserver with Master: 1f72db17759247f1bb122627e07d7427 (127.3.137.66:34431)
I20260809 06:20:18.692121 13694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.66:46221
W20260809 06:20:19.015249 14026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:19.015555 14026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:19.015717 14026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:19.022778 14026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:19.022907 14026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:19.023093 14026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:20:19.032063 14026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:19.033285 14026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:19.035068 14026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:19.048943 14033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:19.049113 14032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:19.050107 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:19.051084 14026 server_base.cc:1034] running on GCE node
I20260809 06:20:19.051785 14026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:19.052757 14026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:19.053959 14026 hybrid_clock.cc:648] HybridClock initialized: now 1786256419053913 us; error 31 us; skew 500 ppm
I20260809 06:20:19.054265 14026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:19.056442 14026 webserver.cc:460] Webserver started at http://127.3.137.67:42041/ using document root <none> and password file <none>
I20260809 06:20:19.057117 14026 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:19.057231 14026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:19.057582 14026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:19.060276 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "3327caf3371947f68e23da726f9c3254"
format_stamp: "Formatted at 2026-08-09 06:20:19 on dist-test-slave-2w3w"
server_key: "453d4173e261bd9e328a3240f4b04a30"
server_key_iv: "e6af56b55c76dc3816e1f91343303f72"
server_key_version: "encryptionkey@0"
I20260809 06:20:19.061061 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "3327caf3371947f68e23da726f9c3254"
format_stamp: "Formatted at 2026-08-09 06:20:19 on dist-test-slave-2w3w"
server_key: "453d4173e261bd9e328a3240f4b04a30"
server_key_iv: "e6af56b55c76dc3816e1f91343303f72"
server_key_version: "encryptionkey@0"
I20260809 06:20:19.065908 14026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 06:20:19.069212 14041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:19.070472 14026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:20:19.070660 14026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "3327caf3371947f68e23da726f9c3254"
format_stamp: "Formatted at 2026-08-09 06:20:19 on dist-test-slave-2w3w"
server_key: "453d4173e261bd9e328a3240f4b04a30"
server_key_iv: "e6af56b55c76dc3816e1f91343303f72"
server_key_version: "encryptionkey@0"
I20260809 06:20:19.070865 14026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:19.084663 14026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:19.085414 14026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:19.085711 14026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:19.086712 14026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:19.088169 14026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:19.088292 14026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:19.088421 14026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:19.088505 14026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:19.121733 14026 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:40739
I20260809 06:20:19.121798 14157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:40739 every 8 connection(s)
I20260809 06:20:19.123160 14026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:20:19.133586 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14026
I20260809 06:20:19.133742 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:20:19.134375 3621 external_mini_cluster.cc:1428] Setting key 6f176b59c84b97b418a0186ade9a601a
I20260809 06:20:19.135154 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:19.125585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:38137 (local address 127.3.137.126:32955)
0809 06:20:19.125801 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:19.125807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:19.127126 (+ 1319us) server_negotiation.cc:407] Connection header received
0809 06:20:19.128946 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:19.128959 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:19.129040 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:19.129128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:19.130116 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.130839 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.132336 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.132731 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.132824 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:19.134296 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:19.134301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:19.134304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:19.134322 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:19.134353 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:19.134365 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:19.134454 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:19.134688 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 06:20:19.134775 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:20:19.136327 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:19.126107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:32955 (local address 127.3.137.67:38137)
0809 06:20:19.126940 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:19.127024 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:19.128594 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:19.129322 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:19.129330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:19.129411 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:19.129969 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:19.129985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.131027 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.131031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:19.132213 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:19.132225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.132923 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.132928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:19.133120 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:19.133961 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:19.133964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:19.134014 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:19.134106 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:19.134466 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:19.134478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:19.134487 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 06:20:19.134676 (+ 189us) client_negotiation.cc:240] Negotiation successful
0809 06:20:19.135097 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
I20260809 06:20:19.137639 14158 heartbeater.cc:347] Connected to a master server at 127.3.137.126:32955
I20260809 06:20:19.137920 14158 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:19.138615 14158 heartbeater.cc:511] Master 127.3.137.126:32955 requested a full tablet report, sending...
I20260809 06:20:19.140007 13694 ts_manager.cc:194] Registered new tserver with Master: 3327caf3371947f68e23da726f9c3254 (127.3.137.67:40739)
I20260809 06:20:19.140779 13694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.67:38137
I20260809 06:20:19.149736 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 06:20:19.155853 13890 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:19.154157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.3.137.126:32955)
0809 06:20:19.154315 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:19.154320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:19.154791 (+ 471us) server_negotiation.cc:407] Connection header received
0809 06:20:19.155013 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:19.155019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:19.155096 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:19.155199 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:19.155600 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44292: BlockingRecv error: recv got EOF from 127.0.0.1:44292 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:20:19.156663 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13754
I20260809 06:20:19.164273 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:19.157554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.3.137.65:35105)
0809 06:20:19.158078 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:19.158084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:19.158107 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:20:19.158164 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:19.158170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:19.158343 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:19.158521 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:19.158903 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.159948 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.161180 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.161710 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.161852 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:19.162848 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:19.162856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:19.162868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:19.163465 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:19.163509 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:19.163534 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:19.163657 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:19.163811 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 06:20:19.163954 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":233,"threads_started":1}
I20260809 06:20:19.165362 13882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:19.246707 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13754
I20260809 06:20:19.261821 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13892
I20260809 06:20:19.269340 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:19.262678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.3.137.66:34431)
0809 06:20:19.263289 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:19.263296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:19.263323 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:20:19.263393 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:19.263399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:19.263610 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:19.263739 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:19.264100 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.265111 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.266316 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.266845 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.266939 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:19.267992 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:19.268000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:19.268012 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:19.268565 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:19.268616 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:19.268635 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:19.268721 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:19.268841 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 06:20:19.268992 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":340,"threads_started":1}
I20260809 06:20:19.270574 14018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:19.352095 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13892
I20260809 06:20:19.369789 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14026
I20260809 06:20:19.378836 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:19.370906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.3.137.67:40739)
0809 06:20:19.371593 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:19.371600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:19.371630 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:20:19.371692 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:19.371699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:19.371926 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:19.372118 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:19.372607 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.373648 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.375267 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:19.375970 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:19.376110 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:19.377427 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:19.377435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:19.377448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:19.378034 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:19.378084 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:19.378104 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:19.378194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:19.378363 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 06:20:19.378488 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":384,"threads_started":1}
I20260809 06:20:19.380091 14153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:19.461632 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14026
I20260809 06:20:19.479205 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 13659
I20260809 06:20:19.480391 13724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:19.581842 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 13659
2026-08-09T06:20:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2422 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T06:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:20:19Z Disabled control of system clock
I20260809 06:20:19.660389 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43181
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 06:20:19.983493 14180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:19.983788 14180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:19.983928 14180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:19.991160 14180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:20:19.991262 14180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:19.991344 14180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:19.991446 14180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:20:19.991531 14180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:20:20.000772 14180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:20.002094 14180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:20.003931 14180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:20.015841 14185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:20.015872 14186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:20.016021 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:20.017030 14180 server_base.cc:1034] running on GCE node
I20260809 06:20:20.017845 14180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:20.019318 14180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:20.020546 14180 hybrid_clock.cc:648] HybridClock initialized: now 1786256420020503 us; error 33 us; skew 500 ppm
I20260809 06:20:20.020931 14180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:20.023288 14180 webserver.cc:460] Webserver started at http://127.3.137.126:39391/ using document root <none> and password file <none>
I20260809 06:20:20.023980 14180 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:20.024106 14180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:20.024420 14180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:20.027338 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "77481c31859e4f1abe6173e72efcd7f7"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "208f4c66520e002aed69d1e091bea112"
server_key_iv: "7562788a41dd5d89efe28a0a23e6b1a1"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.028134 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "77481c31859e4f1abe6173e72efcd7f7"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "208f4c66520e002aed69d1e091bea112"
server_key_iv: "7562788a41dd5d89efe28a0a23e6b1a1"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.033370 14180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:20:20.036933 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:20.038162 14180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:20:20.038342 14180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "77481c31859e4f1abe6173e72efcd7f7"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "208f4c66520e002aed69d1e091bea112"
server_key_iv: "7562788a41dd5d89efe28a0a23e6b1a1"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.038558 14180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:20.044200 14180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:20.044971 14180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:20.045255 14180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:20.063346 14180 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:43181
I20260809 06:20:20.063392 14248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:43181 every 8 connection(s)
I20260809 06:20:20.064567 14180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:20:20.068102 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:20.071000 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14180
I20260809 06:20:20.071225 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:20:20.072136 3621 external_mini_cluster.cc:1428] Setting key 0aa5664c78242a00c743fbcabb948b38
I20260809 06:20:20.080169 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:20.074188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.3.137.126:43181)
0809 06:20:20.075214 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:20.075225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:20.075265 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:20:20.076832 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:20.076854 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:20.077231 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:20.077549 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:20.078630 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:20.078639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:20.078651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:20.078670 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:20.078723 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:20.078741 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:20.078934 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:20.079092 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 06:20:20.079372 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":356,"threads_started":1}
I20260809 06:20:20.080802 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7: Bootstrap starting.
I20260809 06:20:20.083730 14249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:20.085047 14249 log.cc:824] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:20.087805 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7: No bootstrap required, opened a new log
I20260809 06:20:20.094949 14249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } }
I20260809 06:20:20.095352 14249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:20.095455 14249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77481c31859e4f1abe6173e72efcd7f7, State: Initialized, Role: FOLLOWER
I20260809 06:20:20.095992 14249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } }
I20260809 06:20:20.096215 14249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:20:20.096362 14249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:20:20.096525 14249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:20.098591 14249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } }
I20260809 06:20:20.099236 14249 leader_election.cc:302] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77481c31859e4f1abe6173e72efcd7f7; no voters:
I20260809 06:20:20.099844 14249 leader_election.cc:288] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:20:20.099990 14255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:20:20.101248 14255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [term 1 LEADER]: Becoming Leader. State: Replica: 77481c31859e4f1abe6173e72efcd7f7, State: Running, Role: LEADER
I20260809 06:20:20.101784 14255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } }
I20260809 06:20:20.102078 14249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:20:20.104856 14256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77481c31859e4f1abe6173e72efcd7f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } } }
I20260809 06:20:20.104856 14257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77481c31859e4f1abe6173e72efcd7f7. Latest consensus state: current_term: 1 leader_uuid: "77481c31859e4f1abe6173e72efcd7f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481c31859e4f1abe6173e72efcd7f7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43181 } } }
I20260809 06:20:20.105352 14256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:20.105444 14257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:20.108088 14263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:20:20.114329 14263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:20:20.123473 14263 catalog_manager.cc:1347] Generated new cluster ID: 8fca75a748294609982cc1d7f6abbed6
I20260809 06:20:20.123593 14263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:20:20.145325 14263 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:20:20.145726 14263 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:20:20.153858 14263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77481c31859e4f1abe6173e72efcd7f7: Generated new TSK 0
I20260809 06:20:20.154626 14263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:20:20.166220 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--builtin_ntp_servers=127.3.137.84:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 06:20:20.495895 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:20.496196 14275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:20.496351 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:20.503744 14275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:20.503885 14275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:20.504058 14275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:20:20.513530 14275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:20.514896 14275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:20.516763 14275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:20.529562 14280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:20.529515 14283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:20.529547 14281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:20.530831 14275 server_base.cc:1034] running on GCE node
I20260809 06:20:20.531443 14275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:20.532387 14275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:20.533627 14275 hybrid_clock.cc:648] HybridClock initialized: now 1786256420533566 us; error 46 us; skew 500 ppm
I20260809 06:20:20.533937 14275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:20.536789 14275 webserver.cc:460] Webserver started at http://127.3.137.65:40541/ using document root <none> and password file <none>
I20260809 06:20:20.537746 14275 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:20.537935 14275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:20.538357 14275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:20.551046 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "f91134dbdec8419cb8aa029293f22da6"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "6bd5a891c48548a10ffdd2a6d0eead80"
server_key_iv: "aff13aebc19cceb29e18a69bac757e1e"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.552282 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "f91134dbdec8419cb8aa029293f22da6"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "6bd5a891c48548a10ffdd2a6d0eead80"
server_key_iv: "aff13aebc19cceb29e18a69bac757e1e"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.560575 14275 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260809 06:20:20.565577 14289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:20.567562 14275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 06:20:20.567745 14275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "f91134dbdec8419cb8aa029293f22da6"
format_stamp: "Formatted at 2026-08-09 06:20:20 on dist-test-slave-2w3w"
server_key: "6bd5a891c48548a10ffdd2a6d0eead80"
server_key_iv: "aff13aebc19cceb29e18a69bac757e1e"
server_key_version: "encryptionkey@0"
I20260809 06:20:20.568066 14275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:20.577574 14275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:20.578747 14275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:20.579102 14275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:20.580533 14275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:20.582854 14275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:20.582947 14275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:20.583102 14275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:20.583184 14275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:20.618126 14275 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:44603
I20260809 06:20:20.618180 14406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:44603 every 8 connection(s)
I20260809 06:20:20.619643 14275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:20:20.626044 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14275
I20260809 06:20:20.626287 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:20:20.627100 3621 external_mini_cluster.cc:1428] Setting key 41ff82bbeeaf628b25d7f88cfac487aa
I20260809 06:20:20.630839 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--builtin_ntp_servers=127.3.137.84:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 06:20:20.638173 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:20.626261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43181 (local address 127.3.137.65:37703)
0809 06:20:20.627353 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:20.627409 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:20.630016 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:20.631217 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:20.631232 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:20.631369 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:20.635475 (+ 4106us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:20.635480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:20.635544 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:20.635684 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:20.636159 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:20.636174 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:20.636185 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 06:20:20.636514 (+ 329us) client_negotiation.cc:240] Negotiation successful
0809 06:20:20.636825 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260809 06:20:20.639045 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:20.625926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:37703 (local address 127.3.137.126:43181)
0809 06:20:20.626457 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:20.626464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:20.630408 (+ 3944us) server_negotiation.cc:407] Connection header received
0809 06:20:20.630485 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:20.630492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:20.630562 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:20.630682 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:20.635895 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:20.635905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:20.635912 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:20.635939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:20.636006 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:20.636026 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:20.636171 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:20.638551 (+ 2380us) server_negotiation.cc:299] Negotiation successful
0809 06:20:20.638637 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260809 06:20:20.640954 14407 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43181
I20260809 06:20:20.641264 14407 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:20.641899 14407 heartbeater.cc:511] Master 127.3.137.126:43181 requested a full tablet report, sending...
I20260809 06:20:20.644021 14213 ts_manager.cc:194] Registered new tserver with Master: f91134dbdec8419cb8aa029293f22da6 (127.3.137.65:44603)
W20260809 06:20:20.972529 14411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:20.972820 14411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:20.972918 14411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:20.980825 14411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:20.980922 14411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:20.981031 14411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:20:20.989753 14411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:20.991096 14411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:20.992808 14411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:21.005956 14416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:21.005914 14417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:21.005890 14419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:21.006925 14411 server_base.cc:1034] running on GCE node
I20260809 06:20:21.007572 14411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:21.008531 14411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:21.009788 14411 hybrid_clock.cc:648] HybridClock initialized: now 1786256421009706 us; error 46 us; skew 500 ppm
I20260809 06:20:21.010146 14411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:21.016291 14411 webserver.cc:460] Webserver started at http://127.3.137.66:33843/ using document root <none> and password file <none>
I20260809 06:20:21.017004 14411 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:21.017122 14411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:21.017452 14411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:21.020480 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "5c7f047620564af195fffe21061bc240"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "36dc687f5174a7ae434e2c2a5bcfcccc"
server_key_iv: "846d7b1ce6d579072e8c00f489b507fa"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.021270 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7f047620564af195fffe21061bc240"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "36dc687f5174a7ae434e2c2a5bcfcccc"
server_key_iv: "846d7b1ce6d579072e8c00f489b507fa"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.026193 14411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:20:21.029644 14425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.030925 14411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:20:21.031164 14411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "5c7f047620564af195fffe21061bc240"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "36dc687f5174a7ae434e2c2a5bcfcccc"
server_key_iv: "846d7b1ce6d579072e8c00f489b507fa"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.031381 14411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:21.036834 14411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:21.037591 14411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:21.037865 14411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:21.038789 14411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:21.040169 14411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:21.040256 14411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.040400 14411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:21.040488 14411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.083860 14411 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:36547
I20260809 06:20:21.083916 14543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:36547 every 8 connection(s)
I20260809 06:20:21.085120 14411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:20:21.089313 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14411
I20260809 06:20:21.089484 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:20:21.089949 3621 external_mini_cluster.cc:1428] Setting key 1cf642557b5e8d846964060071e5e6e6
I20260809 06:20:21.092760 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--builtin_ntp_servers=127.3.137.84:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 06:20:21.093469 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.087647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39667 (local address 127.3.137.126:43181)
0809 06:20:21.087834 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.087838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.089173 (+ 1335us) server_negotiation.cc:407] Connection header received
0809 06:20:21.091117 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.091123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.091233 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.091356 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.092730 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:21.092735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:21.092740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:21.092759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:21.092798 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:21.092811 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:21.092929 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:21.093121 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 06:20:21.093191 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:20:21.094367 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.088123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43181 (local address 127.3.137.66:39667)
0809 06:20:21.089008 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:21.089049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:21.090786 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:21.091613 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:21.091626 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:21.091744 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:21.092445 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:21.092447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:21.092483 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:21.092566 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:21.092905 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:21.092916 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:21.092924 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:20:21.093200 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 06:20:21.093440 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260809 06:20:21.095676 14544 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43181
I20260809 06:20:21.095964 14544 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:21.096499 14544 heartbeater.cc:511] Master 127.3.137.126:43181 requested a full tablet report, sending...
I20260809 06:20:21.097884 14213 ts_manager.cc:194] Registered new tserver with Master: 5c7f047620564af195fffe21061bc240 (127.3.137.66:36547)
W20260809 06:20:21.419302 14547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:21.419610 14547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:21.419772 14547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:21.427013 14547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:21.427193 14547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:21.427357 14547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:20:21.436617 14547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:21.437986 14547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:21.439818 14547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:21.453270 14556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:21.453296 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:21.453337 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:21.453878 14547 server_base.cc:1034] running on GCE node
I20260809 06:20:21.454535 14547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:21.455507 14547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:21.456764 14547 hybrid_clock.cc:648] HybridClock initialized: now 1786256421456695 us; error 55 us; skew 500 ppm
I20260809 06:20:21.457064 14547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:20:21.459309 14547 webserver.cc:460] Webserver started at http://127.3.137.67:39491/ using document root <none> and password file <none>
I20260809 06:20:21.460003 14547 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:21.460245 14547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:21.460605 14547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:21.463449 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "8803f79936804281ad4c9a7bc50ae35b"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "6f4cafed11d3e95cbd706e02a3d41877"
server_key_iv: "78657cbf92f4e18f9d0059aee3ada33d"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.464234 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "8803f79936804281ad4c9a7bc50ae35b"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "6f4cafed11d3e95cbd706e02a3d41877"
server_key_iv: "78657cbf92f4e18f9d0059aee3ada33d"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.469255 14547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:20:21.472556 14563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.473845 14547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:20:21.474035 14547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "8803f79936804281ad4c9a7bc50ae35b"
format_stamp: "Formatted at 2026-08-09 06:20:21 on dist-test-slave-2w3w"
server_key: "6f4cafed11d3e95cbd706e02a3d41877"
server_key_iv: "78657cbf92f4e18f9d0059aee3ada33d"
server_key_version: "encryptionkey@0"
I20260809 06:20:21.474254 14547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:21.479803 14547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:21.480537 14547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:21.480803 14547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:21.481652 14547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:21.482939 14547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:21.483098 14547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.483224 14547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:21.483323 14547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:21.516330 14547 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:46701
I20260809 06:20:21.516386 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:46701 every 8 connection(s)
I20260809 06:20:21.517624 14547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:20:21.518673 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14547
I20260809 06:20:21.518874 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:20:21.519372 3621 external_mini_cluster.cc:1428] Setting key 456685c73bf9c376975a442889fe325d
I20260809 06:20:21.526304 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.520366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:60893 (local address 127.3.137.126:43181)
0809 06:20:21.520574 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.520581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.522353 (+ 1772us) server_negotiation.cc:407] Connection header received
0809 06:20:21.524101 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.524107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.524182 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.524307 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.525516 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:21.525522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:21.525527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:21.525544 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:21.525583 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:21.525595 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:21.525711 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:21.525956 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 06:20:21.526027 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:20:21.527228 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.521069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43181 (local address 127.3.137.67:60893)
0809 06:20:21.522205 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:21.522248 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:21.523833 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:21.524525 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:21.524533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:21.524619 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:21.525240 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:20:21.525242 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:21.525279 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:20:21.525362 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:21.525686 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:21.525697 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:21.525704 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:20:21.526037 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 06:20:21.526289 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":400,"threads_started":1}
I20260809 06:20:21.528506 14683 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43181
I20260809 06:20:21.528853 14683 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:21.529388 14683 heartbeater.cc:511] Master 127.3.137.126:43181 requested a full tablet report, sending...
I20260809 06:20:21.530695 14213 ts_manager.cc:194] Registered new tserver with Master: 8803f79936804281ad4c9a7bc50ae35b (127.3.137.67:46701)
I20260809 06:20:21.534667 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 06:20:21.540767 14409 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:20:21.539247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.3.137.126:43181)
0809 06:20:21.539429 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.539435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.539972 (+ 537us) server_negotiation.cc:407] Connection header received
0809 06:20:21.540109 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.540112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.540160 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.540230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.540563 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47018: BlockingRecv error: recv got EOF from 127.0.0.1:47018 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:20:21.541165 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14275
I20260809 06:20:21.544924 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.542243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.3.137.65:44603)
0809 06:20:21.542789 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.542798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.542840 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:20:21.542911 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.542917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.543148 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.543306 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.544243 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:21.544249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:21.544258 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:21.544283 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:21.544321 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:21.544335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:21.544471 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:21.544576 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 06:20:21.544671 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260809 06:20:21.546041 14402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:21.628635 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14275
I20260809 06:20:21.647565 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14411
I20260809 06:20:21.650998 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.648430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.3.137.66:36547)
0809 06:20:21.649004 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.649011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.649037 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:20:21.649102 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.649107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.649313 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.649441 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.650276 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:21.650283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:21.650294 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:21.650313 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:21.650354 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:21.650369 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:21.650471 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:21.650581 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 06:20:21.650658 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":294,"threads_started":1}
I20260809 06:20:21.652195 14539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:21.732561 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14411
I20260809 06:20:21.750522 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14547
I20260809 06:20:21.754311 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:21.751545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.3.137.67:46701)
0809 06:20:21.752209 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:21.752217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:21.752249 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:20:21.752312 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:21.752318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:21.752528 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:21.752657 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:21.753571 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:21.753582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:21.753601 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:20:21.753622 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:21.753685 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:20:21.753709 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:21.753902 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:21.753959 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 06:20:21.754044 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":333,"threads_started":1}
I20260809 06:20:21.755676 14678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:21.845026 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14547
I20260809 06:20:21.862944 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14180
I20260809 06:20:21.864476 14244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:21.973836 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14180
2026-08-09T06:20:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2392 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:20:22 dist-test-slave-2w3w krb5kdc[14702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:20:22 dist-test-slave-2w3w krb5kdc[14702](info): set up 2 sockets
Aug 09 06:20:22 dist-test-slave-2w3w krb5kdc[14702](info): commencing operation
krb5kdc: starting...
W20260809 06:20:23.133771 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:20:23 dist-test-slave-2w3w krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:20:23.388471 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46657
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:20:23.716648 14715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:23.716924 14715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:23.717105 14715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:23.724251 14715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:20:23.724386 14715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:23.724498 14715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:23.724596 14715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:20:23.724682 14715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:20:23.734244 14715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:46657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:23.735561 14715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:23.736860 14715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:23.748463 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:23.748572 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:23.748538 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:23.749516 14715 server_base.cc:1034] running on GCE node
I20260809 06:20:23.750231 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:23.751577 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:23.752775 14715 hybrid_clock.cc:648] HybridClock initialized: now 1786256423752737 us; error 28 us; skew 500 ppm
I20260809 06:20:23.753101 14715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:23 dist-test-slave-2w3w krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:23.756155 14715 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:20:23.757602 14715 webserver.cc:460] Webserver started at http://127.3.137.126:40877/ using document root <none> and password file <none>
I20260809 06:20:23.758235 14715 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:23.758375 14715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:23.758680 14715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:23.761749 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "b8076c32772e48b5a57a11ff3c3c7613"
format_stamp: "Formatted at 2026-08-09 06:20:23 on dist-test-slave-2w3w"
server_key: "dd6f63ab23df87bff7506eca5dcaf035"
server_key_iv: "caa4a365056875f9b7361902b909ef97"
server_key_version: "encryptionkey@0"
I20260809 06:20:23.762497 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "b8076c32772e48b5a57a11ff3c3c7613"
format_stamp: "Formatted at 2026-08-09 06:20:23 on dist-test-slave-2w3w"
server_key: "dd6f63ab23df87bff7506eca5dcaf035"
server_key_iv: "caa4a365056875f9b7361902b909ef97"
server_key_version: "encryptionkey@0"
I20260809 06:20:23.767395 14715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 06:20:23.770608 14730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:23.771836 14715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:20:23.772014 14715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "b8076c32772e48b5a57a11ff3c3c7613"
format_stamp: "Formatted at 2026-08-09 06:20:23 on dist-test-slave-2w3w"
server_key: "dd6f63ab23df87bff7506eca5dcaf035"
server_key_iv: "caa4a365056875f9b7361902b909ef97"
server_key_version: "encryptionkey@0"
I20260809 06:20:23.772199 14715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:23.794318 14715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:23.798388 14715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:23.798694 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:23.816954 14715 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:46657
I20260809 06:20:23.816979 14782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:46657 every 8 connection(s)
I20260809 06:20:23.818285 14715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:20:23.821866 14783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:23.824259 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14715
I20260809 06:20:23.824497 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:20:23.825172 3621 external_mini_cluster.cc:1428] Setting key f745498109f5ad95dd7a44e077e0da1f
I20260809 06:20:23.833110 14783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: Bootstrap starting.
I20260809 06:20:23.836287 14783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:23.837280 14783 log.cc:824] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:20:23 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:20:23.840808 14783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: No bootstrap required, opened a new log
I20260809 06:20:23.847683 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:23.827296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.3.137.126:46657)
0809 06:20:23.828060 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:23.828073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:23.828112 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:20:23.829701 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:23.829722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:23.830098 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:23.830502 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:23.832064 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:23.833434 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:23.834724 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:23.835438 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:23.835627 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:23.840504 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:23.840511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:23.840524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:23.840556 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:23.843682 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:23.844473 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:23.844479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:23.844485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:23.844649 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:23.845064 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:23.845068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:23.845069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:23.845562 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:23.845801 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:23.846230 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 06:20:23.846604 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":354,"threads_started":1}
I20260809 06:20:23.849692 14783 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } }
I20260809 06:20:23.850049 14783 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:23.850188 14783 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8076c32772e48b5a57a11ff3c3c7613, State: Initialized, Role: FOLLOWER
I20260809 06:20:23.850745 14783 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } }
I20260809 06:20:23.850950 14783 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:20:23.851115 14783 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:20:23.851244 14783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:23.853353 14783 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } }
I20260809 06:20:23.853775 14783 leader_election.cc:302] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8076c32772e48b5a57a11ff3c3c7613; no voters:
I20260809 06:20:23.854279 14783 leader_election.cc:288] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:20:23.854426 14789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:20:23.855255 14789 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [term 1 LEADER]: Becoming Leader. State: Replica: b8076c32772e48b5a57a11ff3c3c7613, State: Running, Role: LEADER
I20260809 06:20:23.855815 14789 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } }
I20260809 06:20:23.856389 14783 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:20:23.858747 14790 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8076c32772e48b5a57a11ff3c3c7613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } } }
I20260809 06:20:23.859161 14790 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:23.859062 14791 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8076c32772e48b5a57a11ff3c3c7613. Latest consensus state: current_term: 1 leader_uuid: "b8076c32772e48b5a57a11ff3c3c7613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8076c32772e48b5a57a11ff3c3c7613" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46657 } } }
I20260809 06:20:23.859470 14791 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613 [sys.catalog]: This master's current role is: LEADER
I20260809 06:20:23.861524 14798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:20:23.866009 14798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:20:23.873795 14798 catalog_manager.cc:1347] Generated new cluster ID: ea89a4444a8d4222b950c0cd24156f03
I20260809 06:20:23.873905 14798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:20:23.893563 14798 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:20:23.895308 14798 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:20:23.911903 14798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: Generated new TSK 0
I20260809 06:20:23.912743 14798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:20:23.982560 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:20:24.337921 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:24.338186 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:24.338277 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:24.345290 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:24.345391 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:24.345497 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:20:24.355325 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:24.356482 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:24.357743 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:24.370919 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:24.371007 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:24.370973 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:24.372211 14813 server_base.cc:1034] running on GCE node
I20260809 06:20:24.372776 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:24.373660 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:24.374851 14813 hybrid_clock.cc:648] HybridClock initialized: now 1786256424374792 us; error 51 us; skew 500 ppm
I20260809 06:20:24.375168 14813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:24 dist-test-slave-2w3w krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:24.378325 14813 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:20:24.379752 14813 webserver.cc:460] Webserver started at http://127.3.137.65:42887/ using document root <none> and password file <none>
I20260809 06:20:24.380396 14813 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:24.380546 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:24.380861 14813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:24.383692 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "4ec609e6921f40b48f445023d5a79f1a"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "bc187c0068b426a382833bf3eff35d74"
server_key_iv: "cf8825bdb38c49acf9f1b1e311e92639"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.384460 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "4ec609e6921f40b48f445023d5a79f1a"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "bc187c0068b426a382833bf3eff35d74"
server_key_iv: "cf8825bdb38c49acf9f1b1e311e92639"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.389320 14813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 06:20:24.392634 14828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:24.393872 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:20:24.394043 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "4ec609e6921f40b48f445023d5a79f1a"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "bc187c0068b426a382833bf3eff35d74"
server_key_iv: "cf8825bdb38c49acf9f1b1e311e92639"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.394241 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:24.422051 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:24.426055 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:24.426378 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:24.427366 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:24.428656 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:24.428761 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:24.428912 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:24.428995 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:24.462116 14813 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:43463
I20260809 06:20:24.462155 14948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:43463 every 8 connection(s)
I20260809 06:20:24.463470 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:20:24.464260 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14813
I20260809 06:20:24.464514 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:20:24.465485 3621 external_mini_cluster.cc:1428] Setting key 9632562a429e0c89a8a911d9c5d9775e
Aug 09 06:20:24 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:20:24.485265 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:24.466421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:34781 (local address 127.3.137.126:46657)
0809 06:20:24.466912 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:24.466918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:24.467978 (+ 1060us) server_negotiation.cc:407] Connection header received
0809 06:20:24.469697 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:24.469701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:24.469802 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:24.469920 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:24.471401 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:24.472098 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:24.473621 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:24.474035 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:24.474166 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:24.478128 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:24.478135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:24.478140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:24.478184 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:24.481822 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:24.482531 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:24.482539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:24.482545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:24.482702 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:24.483190 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:24.483198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:24.483204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:24.483601 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:24.483754 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:24.484503 (+ 749us) server_negotiation.cc:299] Negotiation successful
0809 06:20:24.484791 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260809 06:20:24.485894 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:24.467065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46657 (local address 127.3.137.65:34781)
0809 06:20:24.467844 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:24.467880 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:24.469459 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:24.470155 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:24.470162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:24.470645 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:24.471243 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:24.471261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:24.472267 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:24.472272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:24.473464 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:24.473481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:24.474264 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:24.474272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:24.474502 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:24.475229 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:24.475250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:24.477914 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:24.481971 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:24.481977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:24.481988 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:24.482396 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:24.482810 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:24.482814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:24.482816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:24.483049 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:24.483775 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:24.483781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:24.484093 (+ 312us) client_negotiation.cc:769] Sending connection context
0809 06:20:24.484321 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 06:20:24.484674 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":344,"threads_started":1}
I20260809 06:20:24.487458 14949 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46657
I20260809 06:20:24.487777 14949 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:24.488622 14949 heartbeater.cc:511] Master 127.3.137.126:46657 requested a full tablet report, sending...
I20260809 06:20:24.490938 14747 ts_manager.cc:194] Registered new tserver with Master: 4ec609e6921f40b48f445023d5a79f1a (127.3.137.65:43463)
I20260809 06:20:24.492482 14747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:34781
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:20:24.530105 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:20:24.852227 14955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:24.852535 14955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:24.852692 14955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:24.860030 14955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:24.860183 14955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:24.860389 14955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:20:24.870097 14955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:24.871459 14955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:24.872749 14955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:24.885619 14961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:24.885784 14963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:24.885808 14960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:24.886935 14955 server_base.cc:1034] running on GCE node
I20260809 06:20:24.887609 14955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:24.888515 14955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:24.889760 14955 hybrid_clock.cc:648] HybridClock initialized: now 1786256424889718 us; error 47 us; skew 500 ppm
I20260809 06:20:24.890087 14955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:24 dist-test-slave-2w3w krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:24.893256 14955 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:20:24.894805 14955 webserver.cc:460] Webserver started at http://127.3.137.66:39299/ using document root <none> and password file <none>
I20260809 06:20:24.895545 14955 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:24.895700 14955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:24.896003 14955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:24.899197 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "94c38f2aaf314e9691502ea880a69321"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "9977348e11b57df5d4ca93e0c2a03af7"
server_key_iv: "438b412343a59d0e44be6b46857b803a"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.900080 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "94c38f2aaf314e9691502ea880a69321"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "9977348e11b57df5d4ca93e0c2a03af7"
server_key_iv: "438b412343a59d0e44be6b46857b803a"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.905493 14955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 06:20:24.908916 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:24.910262 14955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:20:24.910434 14955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "94c38f2aaf314e9691502ea880a69321"
format_stamp: "Formatted at 2026-08-09 06:20:24 on dist-test-slave-2w3w"
server_key: "9977348e11b57df5d4ca93e0c2a03af7"
server_key_iv: "438b412343a59d0e44be6b46857b803a"
server_key_version: "encryptionkey@0"
I20260809 06:20:24.910629 14955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:24.941509 14955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:24.947556 14955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:24.947911 14955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:24.949146 14955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:24.951010 14955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:24.951110 14955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 06:20:24.951237 14955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:24.951305 14955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:24.989734 14955 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41349
I20260809 06:20:24.989743 15090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41349 every 8 connection(s)
I20260809 06:20:24.991706 14955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:20:24.995548 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 14955
I20260809 06:20:24.995796 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:20:24.996488 3621 external_mini_cluster.cc:1428] Setting key b35d1ea43b9f57dffee0b9cae88a10dd
Aug 09 06:20:25 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:20:25.015476 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:24.995000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:53975 (local address 127.3.137.126:46657)
0809 06:20:24.997560 (+ 2560us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:24.997566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:24.997585 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:20:24.999328 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:24.999333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:24.999417 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:24.999538 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:25.001006 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.002195 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.003877 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.004602 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.004747 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.008370 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:25.008379 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:25.008386 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:25.008442 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:25.011757 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.012733 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.012737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.012739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.012867 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.013352 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.013355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.013357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.013631 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:25.013792 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.014758 (+ 966us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.015113 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2445,"thread_start_us":2320,"threads_started":1}
I20260809 06:20:25.016805 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:24.995648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46657 (local address 127.3.137.66:53975)
0809 06:20:24.996564 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:24.996618 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:24.999014 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:24.999698 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:24.999706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.000277 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:25.000831 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.000848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.002339 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.002342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.003728 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.003744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.004801 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.004805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.004994 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.005688 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:25.005710 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:25.008137 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:25.011911 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:25.011937 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:25.011957 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:25.012609 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:25.012977 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:25.012984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:25.012991 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:25.013241 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:25.013758 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:25.013769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:25.014286 (+ 517us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.014621 (+ 335us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.015271 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":377,"threads_started":1}
I20260809 06:20:25.018306 15091 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46657
I20260809 06:20:25.018633 15091 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:25.019486 15091 heartbeater.cc:511] Master 127.3.137.126:46657 requested a full tablet report, sending...
I20260809 06:20:25.020962 14747 ts_manager.cc:194] Registered new tserver with Master: 94c38f2aaf314e9691502ea880a69321 (127.3.137.66:41349)
I20260809 06:20:25.021772 14747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:53975
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:20:25.060218 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:20:25.382690 15098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:20:25.382933 15098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:20:25.383101 15098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:20:25.390044 15098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:20:25.390139 15098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:20:25.390246 15098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:20:25.399950 15098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:20:25.401216 15098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:20:25.402523 15098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:20:25.415714 15103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:25.415743 15106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:20:25.415804 15104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:20:25.417212 15098 server_base.cc:1034] running on GCE node
I20260809 06:20:25.417837 15098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:20:25.418752 15098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:20:25.419981 15098 hybrid_clock.cc:648] HybridClock initialized: now 1786256425419913 us; error 58 us; skew 500 ppm
I20260809 06:20:25.420295 15098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:20:25 dist-test-slave-2w3w krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:20:25.423454 15098 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:20:25.424943 15098 webserver.cc:460] Webserver started at http://127.3.137.67:39959/ using document root <none> and password file <none>
I20260809 06:20:25.425642 15098 fs_manager.cc:359] Metadata directory not provided
I20260809 06:20:25.425796 15098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:20:25.426106 15098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:20:25.429075 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "cccf6e7a95a84815883ab90b5605a28e"
format_stamp: "Formatted at 2026-08-09 06:20:25 on dist-test-slave-2w3w"
server_key: "37e81bdcfdddf2fc77f6e87644499799"
server_key_iv: "a570a285456c8ccf02bfa01fdea7af76"
server_key_version: "encryptionkey@0"
I20260809 06:20:25.429890 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "cccf6e7a95a84815883ab90b5605a28e"
format_stamp: "Formatted at 2026-08-09 06:20:25 on dist-test-slave-2w3w"
server_key: "37e81bdcfdddf2fc77f6e87644499799"
server_key_iv: "a570a285456c8ccf02bfa01fdea7af76"
server_key_version: "encryptionkey@0"
I20260809 06:20:25.435178 15098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:20:25.438661 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:25.440052 15098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:20:25.440351 15098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "cccf6e7a95a84815883ab90b5605a28e"
format_stamp: "Formatted at 2026-08-09 06:20:25 on dist-test-slave-2w3w"
server_key: "37e81bdcfdddf2fc77f6e87644499799"
server_key_iv: "a570a285456c8ccf02bfa01fdea7af76"
server_key_version: "encryptionkey@0"
I20260809 06:20:25.440558 15098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:20:25.454368 15098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:20:25.459062 15098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:20:25.459343 15098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:20:25.460383 15098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:20:25.461879 15098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:20:25.461977 15098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:25.462133 15098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:20:25.462214 15098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:20:25.495956 14949 heartbeater.cc:503] Master 127.3.137.126:46657 was elected leader, sending a full tablet report...
I20260809 06:20:25.499394 15098 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:43529
I20260809 06:20:25.499415 15233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:43529 every 8 connection(s)
I20260809 06:20:25.500658 15098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:20:25.504177 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 15098
I20260809 06:20:25.504383 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:20:25.505013 3621 external_mini_cluster.cc:1428] Setting key 1dc231f6d7f7d8d65ddcc25c6e63bdb3
Aug 09 06:20:25 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:20:25.521842 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.503728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:44253 (local address 127.3.137.126:46657)
0809 06:20:25.503946 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.503952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.505331 (+ 1379us) server_negotiation.cc:407] Connection header received
0809 06:20:25.507746 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.507751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.507870 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.507978 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:25.509467 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.510216 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.511959 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.512362 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.512459 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.515897 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:25.515902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:25.515905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:25.515936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:25.518780 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.519770 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.519776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.519779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.519941 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.520474 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.520478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.520480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.520737 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:25.520864 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.521431 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.521593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:20:25.522879 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.504227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46657 (local address 127.3.137.67:44253)
0809 06:20:25.505131 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.505189 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.507458 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.508234 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.508243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.508726 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:20:25.509282 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.509298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.510402 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.510406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.511831 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.511843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.512545 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.512550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.512738 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.513347 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:20:25.513369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:20:25.515703 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:20:25.519009 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:25.519022 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:25.519037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:25.519632 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:25.520059 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:20:25.520065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:20:25.520070 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:20:25.520341 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:20:25.520839 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:20:25.520845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:20:25.521131 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.521336 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.521708 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":337,"threads_started":1}
I20260809 06:20:25.524215 15234 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46657
I20260809 06:20:25.524514 15234 heartbeater.cc:464] Registering TS with master...
I20260809 06:20:25.525245 15234 heartbeater.cc:511] Master 127.3.137.126:46657 requested a full tablet report, sending...
I20260809 06:20:25.526553 14747 ts_manager.cc:194] Registered new tserver with Master: cccf6e7a95a84815883ab90b5605a28e (127.3.137.67:43529)
I20260809 06:20:25.527391 14747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:44253
I20260809 06:20:25.532262 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:20:25.548700 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.536730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.3.137.126:46657)
0809 06:20:25.536888 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.536893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.537353 (+ 460us) server_negotiation.cc:407] Connection header received
0809 06:20:25.537477 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.537480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.537535 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.537609 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:25.538904 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.539632 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.540752 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.541178 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.541274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.543429 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:25.543434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:25.543437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:25.543465 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:25.546054 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.546729 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.546740 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.546742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.546882 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:25.547256 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:25.547260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:25.547265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:25.547563 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:25.547842 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.548169 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.548380 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:20:25.552521 14747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:20:25.559917 14747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:20:25.582871 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.571856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.3.137.65:43463)
0809 06:20:25.572667 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.572675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.573213 (+ 538us) server_negotiation.cc:407] Connection header received
0809 06:20:25.573567 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.573580 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.573909 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.574099 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:20:25.575412 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.577423 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.580265 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.581712 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.581854 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.582096 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.582375 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.582537 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":459,"threads_started":1}
I20260809 06:20:25.583194 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.571655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43463 (local address 127.0.0.1:58576)
0809 06:20:25.573060 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.573112 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.573309 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.574473 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.574485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.574527 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:20:25.575156 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.575180 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.577596 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.577599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.580055 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.580072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.581919 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.581925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.582131 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.582182 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.582325 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.582507 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":1,"thread_start_us":1075,"threads_started":1}
I20260809 06:20:25.584240 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.571980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41349 (local address 127.0.0.1:51128)
0809 06:20:25.573060 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.573111 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.573295 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.576852 (+ 3557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.576857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.576882 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:20:25.577409 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.577422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.578751 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.578757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.580900 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.580915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.583059 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.583067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.583248 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.583296 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.583389 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.583522 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":54,"thread_start_us":846,"threads_started":1}
I20260809 06:20:25.584244 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.572552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.3.137.66:41349)
0809 06:20:25.576162 (+ 3610us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.576170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.576197 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:20:25.576292 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.576298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.576516 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.576693 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:20:25.577555 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.578615 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.581062 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.582488 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.582634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.582828 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.583720 (+ 892us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.583893 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3417,"thread_start_us":3280,"threads_started":1}
I20260809 06:20:25.585265 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.574152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.3.137.67:43529)
0809 06:20:25.574916 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.574927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.576182 (+ 1255us) server_negotiation.cc:407] Connection header received
0809 06:20:25.576303 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.576308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.576585 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.576779 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:20:25.577857 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.579504 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.581758 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.583235 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.583377 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.583613 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.584760 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.584922 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":407,"threads_started":1}
I20260809 06:20:25.585754 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.573534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43529 (local address 127.0.0.1:48368)
0809 06:20:25.575884 (+ 2350us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.575901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.576069 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.577062 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.577066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.577089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:20:25.577614 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.577631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.579669 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.579675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.581580 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.581592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.583746 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.583753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.583947 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.583983 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.584079 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.585309 (+ 1230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2200,"mutex_wait_us":1,"thread_start_us":1995,"threads_started":1}
I20260809 06:20:25.587555 14877 tablet_service.cc:1467] Processing CreateTablet for tablet ba80328b9da143e6a4d445fab84ffb00 (DEFAULT_TABLE table=test-table [id=688b1f1495634bb2ab0257364026826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:20:25.588078 15021 tablet_service.cc:1467] Processing CreateTablet for tablet ba80328b9da143e6a4d445fab84ffb00 (DEFAULT_TABLE table=test-table [id=688b1f1495634bb2ab0257364026826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:20:25.588568 15166 tablet_service.cc:1467] Processing CreateTablet for tablet ba80328b9da143e6a4d445fab84ffb00 (DEFAULT_TABLE table=test-table [id=688b1f1495634bb2ab0257364026826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:20:25.589283 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba80328b9da143e6a4d445fab84ffb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:25.589895 15166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba80328b9da143e6a4d445fab84ffb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:25.590123 15021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba80328b9da143e6a4d445fab84ffb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:20:25.601914 15253 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Bootstrap starting.
I20260809 06:20:25.604544 15254 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Bootstrap starting.
I20260809 06:20:25.606164 15255 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Bootstrap starting.
I20260809 06:20:25.606902 15253 tablet_bootstrap.cc:654] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:25.608394 15253 log.cc:824] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:25.608484 15254 tablet_bootstrap.cc:654] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:25.609730 15254 log.cc:824] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:25.609951 15255 tablet_bootstrap.cc:654] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Neither blocks nor log segments found. Creating new log.
I20260809 06:20:25.611253 15255 log.cc:824] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:20:25.612144 15253 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: No bootstrap required, opened a new log
I20260809 06:20:25.612444 15253 ts_tablet_manager.cc:1397] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260809 06:20:25.613011 15254 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: No bootstrap required, opened a new log
I20260809 06:20:25.613307 15254 ts_tablet_manager.cc:1397] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 06:20:25.621073 15255 tablet_bootstrap.cc:492] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: No bootstrap required, opened a new log
I20260809 06:20:25.621330 15255 ts_tablet_manager.cc:1397] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260809 06:20:25.624697 15253 raft_consensus.cc:348] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.625196 15253 raft_consensus.cc:374] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:25.625321 15253 raft_consensus.cc:729] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec609e6921f40b48f445023d5a79f1a, State: Initialized, Role: FOLLOWER
I20260809 06:20:25.625103 15254 raft_consensus.cc:348] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.625605 15254 raft_consensus.cc:374] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:25.625710 15254 raft_consensus.cc:729] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c38f2aaf314e9691502ea880a69321, State: Initialized, Role: FOLLOWER
I20260809 06:20:25.626042 15253 consensus_queue.cc:260] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.626322 15254 consensus_queue.cc:260] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.629544 15255 raft_consensus.cc:348] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.629915 15253 ts_tablet_manager.cc:1428] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260809 06:20:25.629974 15255 raft_consensus.cc:374] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:20:25.630071 15255 raft_consensus.cc:729] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccf6e7a95a84815883ab90b5605a28e, State: Initialized, Role: FOLLOWER
I20260809 06:20:25.630718 15255 consensus_queue.cc:260] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.630910 15091 heartbeater.cc:503] Master 127.3.137.126:46657 was elected leader, sending a full tablet report...
I20260809 06:20:25.631585 15254 ts_tablet_manager.cc:1428] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
I20260809 06:20:25.632524 15234 heartbeater.cc:503] Master 127.3.137.126:46657 was elected leader, sending a full tablet report...
I20260809 06:20:25.634378 15255 ts_tablet_manager.cc:1428] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260809 06:20:25.716562 14950 tablet.cc:2369] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:20:25.744105 15093 tablet.cc:2369] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:20:25.752866 15235 tablet.cc:2369] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:20:25.868088 15260 raft_consensus.cc:482] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:20:25.868429 15260 raft_consensus.cc:504] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.870165 15260 leader_election.cc:288] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec609e6921f40b48f445023d5a79f1a (127.3.137.65:43463), cccf6e7a95a84815883ab90b5605a28e (127.3.137.67:43529)
I20260809 06:20:25.877059 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.870543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:60469 (local address 127.3.137.67:43529)
0809 06:20:25.870786 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.870793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.871636 (+ 843us) server_negotiation.cc:407] Connection header received
0809 06:20:25.871882 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.871888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.871962 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.872058 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:20:25.872590 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.873356 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.875197 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.876105 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.876279 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.876363 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.876670 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.876800 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:20:25.877219 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.870800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43529 (local address 127.3.137.66:60469)
0809 06:20:25.871520 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.871543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.871719 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.872078 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.872083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.872109 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:20:25.872448 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.872457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.873490 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.873497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.875054 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.875068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.876356 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.876363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.876519 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.876543 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.876645 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.876770 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":1,"thread_start_us":402,"threads_started":1}
I20260809 06:20:25.877630 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.870920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43463 (local address 127.3.137.66:38057)
0809 06:20:25.871982 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0809 06:20:25.871996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:20:25.872171 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:20:25.872526 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:20:25.872531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:20:25.872557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:20:25.872929 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.872937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.873971 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.873976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.875567 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:20:25.875579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.876850 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.876857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:20:25.877041 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.877067 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:20:25.877191 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 06:20:25.877310 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":710,"threads_started":1}
I20260809 06:20:25.877885 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.870549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:38057 (local address 127.3.137.65:43463)
0809 06:20:25.870786 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.870793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.872099 (+ 1306us) server_negotiation.cc:407] Connection header received
0809 06:20:25.872337 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.872342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.872399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.872499 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:20:25.873088 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.873815 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.875718 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.876642 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.876821 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.876895 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.877509 (+ 614us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.877650 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:20:25.878242 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba80328b9da143e6a4d445fab84ffb00" candidate_uuid: "94c38f2aaf314e9691502ea880a69321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cccf6e7a95a84815883ab90b5605a28e" is_pre_election: true
I20260809 06:20:25.878872 15187 raft_consensus.cc:2454] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c38f2aaf314e9691502ea880a69321 in term 0.
I20260809 06:20:25.878779 14898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba80328b9da143e6a4d445fab84ffb00" candidate_uuid: "94c38f2aaf314e9691502ea880a69321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec609e6921f40b48f445023d5a79f1a" is_pre_election: true
I20260809 06:20:25.879396 14898 raft_consensus.cc:2454] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c38f2aaf314e9691502ea880a69321 in term 0.
I20260809 06:20:25.879655 14971 leader_election.cc:302] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c38f2aaf314e9691502ea880a69321, cccf6e7a95a84815883ab90b5605a28e; no voters:
I20260809 06:20:25.880110 15260 raft_consensus.cc:2781] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:20:25.880230 15260 raft_consensus.cc:482] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:20:25.880306 15260 raft_consensus.cc:3037] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:25.882463 15260 raft_consensus.cc:504] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.883102 15260 leader_election.cc:288] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [CANDIDATE]: Term 1 election: Requested vote from peers 4ec609e6921f40b48f445023d5a79f1a (127.3.137.65:43463), cccf6e7a95a84815883ab90b5605a28e (127.3.137.67:43529)
I20260809 06:20:25.883535 14898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba80328b9da143e6a4d445fab84ffb00" candidate_uuid: "94c38f2aaf314e9691502ea880a69321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec609e6921f40b48f445023d5a79f1a"
I20260809 06:20:25.883648 15187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba80328b9da143e6a4d445fab84ffb00" candidate_uuid: "94c38f2aaf314e9691502ea880a69321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cccf6e7a95a84815883ab90b5605a28e"
I20260809 06:20:25.883754 14898 raft_consensus.cc:3037] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:25.883852 15187 raft_consensus.cc:3037] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:20:25.887071 15187 raft_consensus.cc:2454] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c38f2aaf314e9691502ea880a69321 in term 1.
I20260809 06:20:25.887071 14898 raft_consensus.cc:2454] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c38f2aaf314e9691502ea880a69321 in term 1.
I20260809 06:20:25.887503 14972 leader_election.cc:302] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec609e6921f40b48f445023d5a79f1a, 94c38f2aaf314e9691502ea880a69321; no voters:
I20260809 06:20:25.887816 15260 raft_consensus.cc:2781] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:20:25.888303 15260 raft_consensus.cc:686] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 1 LEADER]: Becoming Leader. State: Replica: 94c38f2aaf314e9691502ea880a69321, State: Running, Role: LEADER
I20260809 06:20:25.888761 15260 consensus_queue.cc:237] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } }
I20260809 06:20:25.893083 14746 catalog_manager.cc:5495] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94c38f2aaf314e9691502ea880a69321 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "94c38f2aaf314e9691502ea880a69321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c38f2aaf314e9691502ea880a69321" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 } health_report { overall_health: UNKNOWN } } }
I20260809 06:20:25.948805 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.943630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.3.137.66:41349)
0809 06:20:25.943822 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.943827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:25.943996 (+ 169us) server_negotiation.cc:407] Connection header received
0809 06:20:25.944170 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:25.944174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:25.944254 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:25.944382 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:20:25.944764 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.945509 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.946881 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:25.947323 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:25.947415 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:25.947831 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:20:25.948331 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:20:25.948440 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:25.948518 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 06:20:25.948582 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:20:25.956763 14898 raft_consensus.cc:1261] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 1 FOLLOWER]: Refusing update from remote peer 94c38f2aaf314e9691502ea880a69321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:20:25.956771 15187 raft_consensus.cc:1261] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 1 FOLLOWER]: Refusing update from remote peer 94c38f2aaf314e9691502ea880a69321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:20:25.957764 15264 consensus_queue.cc:1035] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cccf6e7a95a84815883ab90b5605a28e" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:20:25.958096 15260 consensus_queue.cc:1035] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec609e6921f40b48f445023d5a79f1a" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:20:25.969602 15267 mvcc.cc:205] Tried to move back new op lower bound from 7316506320709279744 to 7316506320443797504. Current Snapshot: MvccSnapshot[applied={T|T < 7316506320709279744}]
I20260809 06:20:25.971032 15268 mvcc.cc:205] Tried to move back new op lower bound from 7316506320709279744 to 7316506320443797504. Current Snapshot: MvccSnapshot[applied={T|T < 7316506320709279744}]
I20260809 06:20:25.983511 14747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:20:25.983803 14747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:20:25.987226 14747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8076c32772e48b5a57a11ff3c3c7613: Sending DeleteTablet for 3 replicas of tablet ba80328b9da143e6a4d445fab84ffb00
I20260809 06:20:25.988471 15021 tablet_service.cc:1514] Processing DeleteTablet for tablet ba80328b9da143e6a4d445fab84ffb00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:20:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:51128
I20260809 06:20:25.988529 14877 tablet_service.cc:1514] Processing DeleteTablet for tablet ba80328b9da143e6a4d445fab84ffb00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:20:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:58576
I20260809 06:20:25.988662 15166 tablet_service.cc:1514] Processing DeleteTablet for tablet ba80328b9da143e6a4d445fab84ffb00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:20:25 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:48368
I20260809 06:20:25.989425 15278 tablet_replica.cc:330] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: stopping tablet replica
I20260809 06:20:25.989887 15278 raft_consensus.cc:2229] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:20:25.990394 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14813
I20260809 06:20:25.990411 15278 raft_consensus.cc:2258] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:20:26.004007 15277 tablet_replica.cc:330] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: stopping tablet replica
I20260809 06:20:26.004490 15277 raft_consensus.cc:2229] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:20:26.004572 15279 tablet_replica.cc:330] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: stopping tablet replica
I20260809 06:20:26.004971 15279 raft_consensus.cc:2229] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:20:26.005270 15277 raft_consensus.cc:2258] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:20:26.005474 15279 raft_consensus.cc:2258] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:20:26.007781 15279 ts_tablet_manager.cc:1905] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:20:26.008750 15278 ts_tablet_manager.cc:1905] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:20:26.008908 15277 ts_tablet_manager.cc:1905] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:20:26.016171 15279 ts_tablet_manager.cc:1918] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:20:26.016438 15279 log.cc:1196] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/ba80328b9da143e6a4d445fab84ffb00
I20260809 06:20:26.017122 15279 ts_tablet_manager.cc:1939] T ba80328b9da143e6a4d445fab84ffb00 P cccf6e7a95a84815883ab90b5605a28e: Deleting consensus metadata
I20260809 06:20:26.017305 15278 ts_tablet_manager.cc:1918] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:20:26.017465 15278 log.cc:1196] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/ba80328b9da143e6a4d445fab84ffb00
I20260809 06:20:26.018031 15278 ts_tablet_manager.cc:1939] T ba80328b9da143e6a4d445fab84ffb00 P 4ec609e6921f40b48f445023d5a79f1a: Deleting consensus metadata
I20260809 06:20:26.019640 14731 catalog_manager.cc:4841] TS cccf6e7a95a84815883ab90b5605a28e (127.3.137.67:43529): tablet ba80328b9da143e6a4d445fab84ffb00 (table test-table [id=688b1f1495634bb2ab0257364026826a]) successfully deleted
I20260809 06:20:26.021704 14732 catalog_manager.cc:4841] TS 4ec609e6921f40b48f445023d5a79f1a (127.3.137.65:43463): tablet ba80328b9da143e6a4d445fab84ffb00 (table test-table [id=688b1f1495634bb2ab0257364026826a]) successfully deleted
I20260809 06:20:26.023490 15277 ts_tablet_manager.cc:1918] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:20:26.023773 15277 log.cc:1196] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/ba80328b9da143e6a4d445fab84ffb00
I20260809 06:20:26.024375 15277 ts_tablet_manager.cc:1939] T ba80328b9da143e6a4d445fab84ffb00 P 94c38f2aaf314e9691502ea880a69321: Deleting consensus metadata
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:20:26.028496 14731 catalog_manager.cc:4841] TS 94c38f2aaf314e9691502ea880a69321 (127.3.137.66:41349): tablet ba80328b9da143e6a4d445fab84ffb00 (table test-table [id=688b1f1495634bb2ab0257364026826a]) successfully deleted
I20260809 06:20:26.036348 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:25.991806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.3.137.65:43463)
0809 06:20:25.992015 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:25.992021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:26.003275 (+ 11254us) server_negotiation.cc:407] Connection header received
0809 06:20:26.003503 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:26.003512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:26.003623 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:26.003734 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:26.012373 (+ 8639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.013609 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.015943 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.016635 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.016739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:26.029321 (+ 12582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:26.029336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:26.029353 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:26.029398 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:26.033262 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.034042 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.034053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.034057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.034233 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.034640 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.034647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.034653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.035173 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:26.035507 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:26.035713 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 06:20:26.035954 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:20:26.037106 14943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:26.152220 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14813
I20260809 06:20:26.173498 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14955
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:20:26.189623 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:26.174409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.3.137.66:41349)
0809 06:20:26.174604 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:26.174613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:26.174635 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:20:26.174718 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:26.174723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:26.174812 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:26.174930 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:26.176384 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.177313 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.178540 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.179143 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.179290 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:26.183668 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:26.183681 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:26.183697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:26.183738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:26.186793 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.187494 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.187504 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.187514 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.187651 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.188083 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.188093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.188095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.188440 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:26.188813 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:26.189031 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 06:20:26.189252 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:20:26.190399 15086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:26.306923 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14955
I20260809 06:20:26.328752 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 15098
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[14702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:20:26.343155 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:20:26.329519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.3.137.67:43529)
0809 06:20:26.329684 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:20:26.329688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:20:26.329706 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:20:26.329853 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:20:26.329857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:20:26.329928 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:20:26.330020 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:20:26.331293 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.332003 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.333196 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:20:26.333574 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:20:26.333665 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:20:26.337895 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:20:26.337908 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:20:26.337922 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:20:26.337962 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:20:26.340775 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.341395 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.341401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.341403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.341546 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:20:26.341861 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:20:26.341864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:20:26.341866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:20:26.342196 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:20:26.342410 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:20:26.342650 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 06:20:26.342809 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:20:26.343809 15229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:26.433624 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 15098
I20260809 06:20:26.451812 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 14715
I20260809 06:20:26.453120 14778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:20:26.561829 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 14715
2026-08-09T06:20:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4589 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[15288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[15288](info): set up 2 sockets
Aug 09 06:20:26 dist-test-slave-2w3w krb5kdc[15288](info): commencing operation
krb5kdc: starting...
W20260809 06:20:27.725915 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:20:27 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:20:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangeradmin_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangeradmin_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangerlookup_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangerlookup_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
I20260809 06:20:28.097851 3621 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Z6xHV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/postgres -l logfile start
2026-08-09 06:20:30.555 UTC [15320] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 06:20:30.555 UTC [15320] LOG: listening on IPv4 address "127.3.137.84", port 33705
2026-08-09 06:20:30.559 UTC [15320] LOG: listening on Unix socket "/tmp/.s.PGSQL.33705"
2026-08-09 06:20:30.579 UTC [15324] LOG: database system was shut down at 2026-08-09 06:20:29 UTC
2026-08-09 06:20:30.584 UTC [15320] LOG: database system is ready to accept connections
I20260809 06:20:31.605661 3621 mini_postgres.cc:96] Postgres bound to 33705
2026-08-09 06:20:31.629 UTC [15332] FATAL: database "slave" does not exist
127.3.137.84:33705 - accepting connections
I20260809 06:20:31.630033 3621 mini_ranger.cc:161] Starting Ranger...
I20260809 06:20:31.659286 3621 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 06:20:32.134936 3621 mini_ranger.cc:87] Created ranger Postgres database
I20260809 06:20:32.135109 3621 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin
2026-08-09 06:20:32,455 [I] DB FLAVOR :POSTGRES
2026-08-09 06:20:32,456 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:20:32,456 [I] Checking connection..
2026-08-09 06:20:32,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 06:20:32,715 [I] Checking connection passed.
2026-08-09 06:20:32,715 [I] --------- Verifying version history table ---------
2026-08-09 06:20:32,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:20:32,974 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:20:32,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:20:33,218 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:20:33,218 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 06:20:33,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 06:20:33.458 UTC [15400] WARNING: there is no transaction in progress
2026-08-09 06:20:33,465 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 06:20:33,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:20:33,727 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 06:20:33,727 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:20:33,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 06:20:33,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 06:20:34,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-09 06:20:34,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:20:34,730 [I] Table x_portal_user does not exist in database ranger
2026-08-09 06:20:34,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:20:34,976 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 06:20:34,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:20:35,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:20:35,517 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 06:20:35,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 06:20:36.473 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.489 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.505 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.520 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.536 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.554 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.608 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.614 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.629 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.639 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.649 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.663 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.672 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.678 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.686 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:36.693 UTC [15513] WARNING: there is no transaction in progress
2026-08-09 06:20:37,309 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 06:20:37,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 06:20:37,547 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 06:20:37,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:20:37,812 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 06:20:37,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:20:38,066 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 06:20:38,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:20:38,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:20:38,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-09 06:20:38,791 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 06:20:38,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 06:20:39,051 [I] DB_PATCHES have already been applied
I20260809 06:20:39.057088 3621 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Z6xHV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 06:20:39.057179 3621 mini_ranger.cc:193] Using host: 127.3.137.84
I20260809 06:20:39.076653 3621 mini_ranger.cc:227] Ranger admin URL: http://127.3.137.84:38991
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 6:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 06:20:40.262686 3621 mini_ranger.cc:160] Time spent starting Ranger: real 8.633s user 0.000s sys 0.054s
Aug 09, 2026 6:21:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 6:21:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 6:21:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 6:21:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 6:21:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 06:21:02.054128 3621 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangerkms_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/rangerkms_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.137.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.137.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 06:21:02 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256462, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 06:21:02.253121 3621 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 06:21:02.278633 3621 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 06:21:03.061115 3621 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 06:21:03.061277 3621 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms
I20260809 06:21:03.061422 3621 mini_ranger_kms.cc:227] Using postgres at 127.3.137.84:33705
2026-08-09 06:21:03,580 [I] DB FLAVOR :POSTGRES
2026-08-09 06:21:03,580 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:21:03,580 [I] Checking connection
2026-08-09 06:21:03,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 06:21:03,826 [I] connection success
2026-08-09 06:21:03,826 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 06:21:03,826 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 06:21:03,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 06:21:04,062 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 06:21:04,063 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:21:04,063 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 06:21:04,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:33705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 06:21:04,347 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 06:21:04.812446 3621 mini_ranger_kms.cc:310] Created kms service
I20260809 06:21:05.007334 3621 mini_ranger_kms.cc:326] Created kudu user
I20260809 06:21:05.054896 3621 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 06:21:05 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256462, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:05.452337 3621 mini_ranger_kms.cc:384] Added ranger policy
I20260809 06:21:05.453670 3621 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 06:21:05.453751 3621 mini_ranger_kms.cc:240] Using host: 127.3.137.84
I20260809 06:21:05.485517 3621 mini_ranger_kms.cc:276] Ranger KMS PID: 15703
I20260809 06:21:05.485699 3621 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.137.84:47871
Aug 09, 2026 6:21:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:21:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:21:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 6:21:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 6:21:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 6:21:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 06:21:07.968235 3621 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.715s user 0.158s sys 0.405s
I20260809 06:21:07.968647 3621 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 06:21:07.968887 3621 mini_ranger_kms.cc:401] 127.3.137.84:47871/kms/v1/keys
Aug 09 06:21:08 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786256468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 06:21:08 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
Aug 09 06:21:08 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
Aug 09 06:21:11 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:21:11.364306 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33565
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:33565
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:47871/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:21:11.715808 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:21:11.716143 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:21:11.716312 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:21:11.723773 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:21:11.723910 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:21:11.724013 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:21:11.724115 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:21:11.724208 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:21:11.734395 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.137.84:47871/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:33565
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:21:11.735780 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:21:11.737074 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:21:11.749871 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:11.749922 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:11.749922 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:21:11.750303 15757 server_base.cc:1034] running on GCE node
I20260809 06:21:11.751231 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:21:11.752614 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:21:11.753818 15757 hybrid_clock.cc:648] HybridClock initialized: now 1786256471753776 us; error 29 us; skew 500 ppm
I20260809 06:21:11.754169 15757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:21:11 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:21:11.761976 15757 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:21:11.763486 15757 webserver.cc:460] Webserver started at http://127.3.137.126:34479/ using document root <none> and password file <none>
I20260809 06:21:11.764107 15757 fs_manager.cc:359] Metadata directory not provided
I20260809 06:21:11.764206 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:21:11.764453 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:21:11 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:11.829345 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "0d268dbb79d445e383ead3c6e7891074"
format_stamp: "Formatted at 2026-08-09 06:21:11 on dist-test-slave-2w3w"
server_key: "305339db2fddd24bfa2831d76c52252d"
server_key_iv: "e573c59f187a059993526397f9153c71"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:11.830217 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "0d268dbb79d445e383ead3c6e7891074"
format_stamp: "Formatted at 2026-08-09 06:21:11 on dist-test-slave-2w3w"
server_key: "305339db2fddd24bfa2831d76c52252d"
server_key_iv: "e573c59f187a059993526397f9153c71"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:11.835883 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:21:11.855880 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:11.857386 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:21:11.857568 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "0d268dbb79d445e383ead3c6e7891074"
format_stamp: "Formatted at 2026-08-09 06:21:11 on dist-test-slave-2w3w"
server_key: "305339db2fddd24bfa2831d76c52252d"
server_key_iv: "e573c59f187a059993526397f9153c71"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:11.857812 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:21:11.888226 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:21:11.892534 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:21:11.892904 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:21:11.914386 15757 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:33565
I20260809 06:21:11.914390 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:33565 every 8 connection(s)
I20260809 06:21:11.915891 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:21:11.917105 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 15757
I20260809 06:21:11.917302 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:21:11.918227 3621 external_mini_cluster.cc:1428] Setting key 1a7913f105f7f861d0021bfd46780f07
I20260809 06:21:11.924615 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:21:11 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:21:11.935302 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: Bootstrap starting.
I20260809 06:21:11.938629 15829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: Neither blocks nor log segments found. Creating new log.
I20260809 06:21:11.939778 15829 log.cc:824] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: Log is configured to *not* fsync() on all Append() calls
I20260809 06:21:11.942814 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: No bootstrap required, opened a new log
I20260809 06:21:11.943384 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:11.920768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.3.137.126:33565)
0809 06:21:11.923624 (+ 2856us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:11.923647 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:11.924245 (+ 598us) server_negotiation.cc:407] Connection header received
0809 06:21:11.926168 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:11.926189 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:11.926549 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:11.926954 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:11.928341 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:11.929738 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:11.931018 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:11.931679 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:11.931930 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:11.935891 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:11.935898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:11.935912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:11.935956 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:11.939237 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:11.939916 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:11.939926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:11.939936 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:11.940125 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:11.940516 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:11.940521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:11.940525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:11.941033 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:11.941332 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:11.941795 (+ 463us) server_negotiation.cc:299] Negotiation successful
0809 06:21:11.942185 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2525,"thread_start_us":2174,"threads_started":1}
I20260809 06:21:11.951615 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } }
I20260809 06:21:11.951974 15829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:21:11.952073 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d268dbb79d445e383ead3c6e7891074, State: Initialized, Role: FOLLOWER
I20260809 06:21:11.952623 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } }
I20260809 06:21:11.952854 15829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:21:11.952991 15829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:21:11.953155 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:21:11.955659 15829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } }
I20260809 06:21:11.956169 15829 leader_election.cc:302] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d268dbb79d445e383ead3c6e7891074; no voters:
I20260809 06:21:11.956791 15829 leader_election.cc:288] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:21:11.956935 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:21:11.957770 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [term 1 LEADER]: Becoming Leader. State: Replica: 0d268dbb79d445e383ead3c6e7891074, State: Running, Role: LEADER
I20260809 06:21:11.958308 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } }
I20260809 06:21:11.958915 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:21:11.961256 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d268dbb79d445e383ead3c6e7891074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } } }
I20260809 06:21:11.961251 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d268dbb79d445e383ead3c6e7891074. Latest consensus state: current_term: 1 leader_uuid: "0d268dbb79d445e383ead3c6e7891074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d268dbb79d445e383ead3c6e7891074" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33565 } } }
I20260809 06:21:11.961742 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [sys.catalog]: This master's current role is: LEADER
I20260809 06:21:11.961844 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074 [sys.catalog]: This master's current role is: LEADER
I20260809 06:21:11.967972 15842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:21:11.973676 15842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:21:11.974901 15829 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:21:11.992161 15842 catalog_manager.cc:1347] Generated new cluster ID: 86d7edbbc98149248cf8a89649a5795f
I20260809 06:21:11.992319 15842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:21:12.033516 15842 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:21:12.035892 15842 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:21:12.048328 15842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: Generated new TSK 0
I20260809 06:21:12.049405 15842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:21:12.844354 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:47871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:21:13 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 06:21:13.294138 15875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:21:13.294482 15875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:21:13.294734 15875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:21:13.302700 15875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:21:13.302866 15875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:21:13.303087 15875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:21:13.313652 15875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.137.84:47871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:21:13.315084 15875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:21:13.316365 15875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:21:13.333191 15885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:13.337728 15886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:21:13.339241 15875 server_base.cc:1034] running on GCE node
W20260809 06:21:13.340219 15889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:21:13.341233 15875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:21:13.348065 15875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:21:13.349414 15875 hybrid_clock.cc:648] HybridClock initialized: now 1786256473349299 us; error 110 us; skew 500 ppm
I20260809 06:21:13.349848 15875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:21:13 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:21:13.357172 15875 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:21:13.361696 15875 webserver.cc:460] Webserver started at http://127.3.137.65:35873/ using document root <none> and password file <none>
I20260809 06:21:13.362928 15875 fs_manager.cc:359] Metadata directory not provided
I20260809 06:21:13.363114 15875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:21:13.363576 15875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:21:13 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:13.387650 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "243f81f49ff64baf832f9291afaeaadd"
format_stamp: "Formatted at 2026-08-09 06:21:13 on dist-test-slave-2w3w"
server_key: "9458f62d73851491f3e91e1276ec41f6"
server_key_iv: "0f71cb70dbdd16f9d0281e3cffc7d4db"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:13.388643 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "243f81f49ff64baf832f9291afaeaadd"
format_stamp: "Formatted at 2026-08-09 06:21:13 on dist-test-slave-2w3w"
server_key: "9458f62d73851491f3e91e1276ec41f6"
server_key_iv: "0f71cb70dbdd16f9d0281e3cffc7d4db"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:13.396936 15875 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260809 06:21:13.418136 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:13.419961 15875 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:21:13.420186 15875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "243f81f49ff64baf832f9291afaeaadd"
format_stamp: "Formatted at 2026-08-09 06:21:13 on dist-test-slave-2w3w"
server_key: "9458f62d73851491f3e91e1276ec41f6"
server_key_iv: "0f71cb70dbdd16f9d0281e3cffc7d4db"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:13.420459 15875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:21:13.461149 15875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:21:13.467144 15875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:21:13.467607 15875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:21:13.475308 15875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:21:13.483603 15875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:21:13.483775 15875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 06:21:13.483987 15875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:21:13.484112 15875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:21:13 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:13.939440 15875 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:38803
I20260809 06:21:13.939568 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:38803 every 8 connection(s)
I20260809 06:21:13.941133 15875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:21:13.951195 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 15875
I20260809 06:21:13.951416 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:21:13.952265 3621 external_mini_cluster.cc:1428] Setting key be72dc0759af3ebbd9c334385cc66bdc
Aug 09 06:21:14 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
I20260809 06:21:14.020548 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:13.945341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:38527 (local address 127.3.137.126:33565)
0809 06:21:13.990014 (+ 44673us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:13.990021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:13.991295 (+ 1274us) server_negotiation.cc:407] Connection header received
0809 06:21:13.999353 (+ 8058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:13.999361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:13.999463 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:13.999562 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:14.002884 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.004130 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.006820 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.007486 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.007588 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:14.012454 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:14.012461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:14.012466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:14.012510 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:14.016449 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:14.017506 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:14.017513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:14.017517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:14.017677 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:14.018278 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:14.018283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:14.018286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:14.018651 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:14.018777 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:14.019842 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0809 06:21:14.020077 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44508,"thread_start_us":44700,"threads_started":1}
I20260809 06:21:14.023048 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:13.944666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33565 (local address 127.3.137.65:38527)
0809 06:21:13.990886 (+ 46220us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:13.990996 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:13.993880 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:14.000206 (+ 6326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:14.000221 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:14.000937 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:21:14.002655 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:14.002681 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.004375 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.004380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:14.006627 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:14.006649 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.007952 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.007960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:14.008278 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:14.009215 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:21:14.009264 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:21:14.012138 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:21:14.016616 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:14.016627 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:14.016647 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:14.017331 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:14.017819 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:14.017824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:14.017828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:14.018118 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:14.018897 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:21:14.018905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:21:14.019386 (+ 481us) client_negotiation.cc:769] Sending connection context
0809 06:21:14.021063 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0809 06:21:14.021626 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45789,"thread_start_us":49442,"threads_started":1}
I20260809 06:21:14.025156 16016 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33565
I20260809 06:21:14.025506 16016 heartbeater.cc:464] Registering TS with master...
I20260809 06:21:14.026427 16016 heartbeater.cc:511] Master 127.3.137.126:33565 requested a full tablet report, sending...
I20260809 06:21:14.029182 15791 ts_manager.cc:194] Registered new tserver with Master: 243f81f49ff64baf832f9291afaeaadd (127.3.137.65:38803)
I20260809 06:21:14.031257 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:38527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
W20260809 06:21:14.062194 3621 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.137.66: real 0.107s user 0.004s sys 0.021s
I20260809 06:21:14.063197 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:47871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:21:14 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
W20260809 06:21:14.511646 16024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:21:14.511977 16024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:21:14.512145 16024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:21:14.519704 16024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:21:14.519860 16024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:21:14.520027 16024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:21:14.530544 16024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.137.84:47871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:21:14.532385 16024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:21:14.533691 16024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:21:14.547248 16037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:14.547259 16038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:14.548110 16040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:21:14.549917 16024 server_base.cc:1034] running on GCE node
I20260809 06:21:14.550560 16024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:21:14.551599 16024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:21:14.552841 16024 hybrid_clock.cc:648] HybridClock initialized: now 1786256474552785 us; error 45 us; skew 500 ppm
I20260809 06:21:14.553207 16024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:21:14 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:21:14.556864 16024 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:21:14.558460 16024 webserver.cc:460] Webserver started at http://127.3.137.66:41263/ using document root <none> and password file <none>
I20260809 06:21:14.559332 16024 fs_manager.cc:359] Metadata directory not provided
I20260809 06:21:14.559479 16024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:21:14.559809 16024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:21:14 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:14.579217 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "5f623160f9e24ec8a43d84551cde0ad1"
format_stamp: "Formatted at 2026-08-09 06:21:14 on dist-test-slave-2w3w"
server_key: "1814f1a6e2965d88a0973ee6e61ba4a2"
server_key_iv: "95f877d7c2599cb536f17373696d59f9"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:14.580533 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "5f623160f9e24ec8a43d84551cde0ad1"
format_stamp: "Formatted at 2026-08-09 06:21:14 on dist-test-slave-2w3w"
server_key: "1814f1a6e2965d88a0973ee6e61ba4a2"
server_key_iv: "95f877d7c2599cb536f17373696d59f9"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:14.588306 16024 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260809 06:21:14.607867 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:14.609740 16024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:21:14.609961 16024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "5f623160f9e24ec8a43d84551cde0ad1"
format_stamp: "Formatted at 2026-08-09 06:21:14 on dist-test-slave-2w3w"
server_key: "1814f1a6e2965d88a0973ee6e61ba4a2"
server_key_iv: "95f877d7c2599cb536f17373696d59f9"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:14.610220 16024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:21:14.633018 16024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:21:14.637470 16024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:21:14.637883 16024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:21:14.639252 16024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:21:14.640882 16024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:21:14.640987 16024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:14.641144 16024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:21:14.641220 16024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:14.681466 16024 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:46701
I20260809 06:21:14.681486 16162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:46701 every 8 connection(s)
I20260809 06:21:14.682827 16024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:21:14.691380 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 16024
I20260809 06:21:14.691552 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:21:14.692080 3621 external_mini_cluster.cc:1428] Setting key 323edb8cc8bc77a28abd14cccc318e88
Aug 09 06:21:14 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:21:14.708272 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:14.685737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:45023 (local address 127.3.137.126:33565)
0809 06:21:14.686228 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:14.686233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:14.687360 (+ 1127us) server_negotiation.cc:407] Connection header received
0809 06:21:14.689272 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:14.689278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:14.689378 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:14.689504 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:14.691083 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.692284 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.694469 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.695250 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.695429 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:14.700107 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:14.700113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:14.700118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:14.700159 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:14.704023 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:14.705013 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:14.705018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:14.705022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:14.705185 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:14.705725 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:14.705742 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:14.705745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:14.706097 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:14.706219 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:14.707340 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0809 06:21:14.707638 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260809 06:21:14.709309 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:14.686354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33565 (local address 127.3.137.66:45023)
0809 06:21:14.687181 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:14.687223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:14.688998 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:14.689754 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:14.689762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:14.690220 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:21:14.690787 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:14.690803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.692463 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.692469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:14.694308 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:14.694323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:14.695517 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:14.695526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:14.695852 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:14.696680 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:21:14.696708 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:21:14.699853 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:21:14.704213 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:14.704223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:14.704241 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:14.704857 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:14.705351 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:14.705355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:14.705359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:14.705590 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:14.706352 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:21:14.706359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:21:14.706832 (+ 473us) client_negotiation.cc:769] Sending connection context
0809 06:21:14.707194 (+ 362us) client_negotiation.cc:240] Negotiation successful
0809 06:21:14.707880 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":308,"threads_started":1}
I20260809 06:21:14.711268 16163 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33565
I20260809 06:21:14.711666 16163 heartbeater.cc:464] Registering TS with master...
I20260809 06:21:14.712584 16163 heartbeater.cc:511] Master 127.3.137.126:33565 requested a full tablet report, sending...
I20260809 06:21:14.714373 15791 ts_manager.cc:194] Registered new tserver with Master: 5f623160f9e24ec8a43d84551cde0ad1 (127.3.137.66:46701)
I20260809 06:21:14.715646 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:45023
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:21:14.757998 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:47871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:21:15.035064 16016 heartbeater.cc:503] Master 127.3.137.126:33565 was elected leader, sending a full tablet report...
W20260809 06:21:15.107196 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:21:15.107475 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:21:15.107658 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:21:15.115083 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:21:15.115208 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:21:15.115414 16170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:21:15.125536 16170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.137.84:47871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:21:15.126791 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:21:15.128083 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:21:15.140908 16175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:15.141014 16178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:21:15.141039 16176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:21:15.141839 16170 server_base.cc:1034] running on GCE node
I20260809 06:21:15.142493 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:21:15.143517 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:21:15.144766 16170 hybrid_clock.cc:648] HybridClock initialized: now 1786256475144714 us; error 48 us; skew 500 ppm
I20260809 06:21:15.145084 16170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:21:15 dist-test-slave-2w3w krb5kdc[15288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256475, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:21:15.151234 16170 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:21:15.152726 16170 webserver.cc:460] Webserver started at http://127.3.137.67:32825/ using document root <none> and password file <none>
I20260809 06:21:15.153378 16170 fs_manager.cc:359] Metadata directory not provided
I20260809 06:21:15.153522 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:21:15.153838 16170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:21:15 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256475, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:15.169864 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "71bb8e45f7e343edb80b4057eedd50f2"
format_stamp: "Formatted at 2026-08-09 06:21:15 on dist-test-slave-2w3w"
server_key: "f6684132c14b8bbd188ff4053b43e148"
server_key_iv: "e672d686464ccf0ae1b48559f4985ef8"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:15.170703 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "71bb8e45f7e343edb80b4057eedd50f2"
format_stamp: "Formatted at 2026-08-09 06:21:15 on dist-test-slave-2w3w"
server_key: "f6684132c14b8bbd188ff4053b43e148"
server_key_iv: "e672d686464ccf0ae1b48559f4985ef8"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:15.176493 16170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:21:15.191969 16185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:15.193580 16170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:21:15.193790 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "71bb8e45f7e343edb80b4057eedd50f2"
format_stamp: "Formatted at 2026-08-09 06:21:15 on dist-test-slave-2w3w"
server_key: "f6684132c14b8bbd188ff4053b43e148"
server_key_iv: "e672d686464ccf0ae1b48559f4985ef8"
server_key_version: "kuduclusterkey@0"
I20260809 06:21:15.194020 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:21:15.232640 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:21:15.236990 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:21:15.237346 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:21:15.238425 16170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:21:15.239956 16170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:21:15.240031 16170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:15.240177 16170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:21:15.240231 16170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:21:15.292150 16170 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:37319
I20260809 06:21:15.292214 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:37319 every 8 connection(s)
I20260809 06:21:15.294070 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:21:15.295481 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 16170
I20260809 06:21:15.295672 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:21:15.296310 3621 external_mini_cluster.cc:1428] Setting key dc426b18eb61a19732a5de2f1169cb62
Aug 09 06:21:15 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256475, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:21:15.318095 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.300426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47379 (local address 127.3.137.126:33565)
0809 06:21:15.300838 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.300843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.301939 (+ 1096us) server_negotiation.cc:407] Connection header received
0809 06:21:15.304420 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.304429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.304545 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.304678 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:15.306231 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.307047 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.308535 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.308971 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.309113 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.312723 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:15.312729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:15.312734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:15.312773 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:15.315419 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.316139 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.316143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.316145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.316268 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.316696 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.316699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.316701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.316948 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:15.317063 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.317673 (+ 610us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.317835 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260809 06:21:15.319068 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.301091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:33565 (local address 127.3.137.67:47379)
0809 06:21:15.301760 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.301795 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.304053 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.304950 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.304963 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.305519 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:21:15.306065 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.306081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.307192 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.307195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.308397 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.308416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.309154 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.309158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.309348 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.310029 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:21:15.310056 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:21:15.312504 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:21:15.315550 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:15.315556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:15.315567 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:15.316056 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:15.316362 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:21:15.316365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:21:15.316367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:21:15.316578 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:21:15.317056 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:21:15.317061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:21:15.317370 (+ 309us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.317590 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.317928 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":268,"threads_started":1}
I20260809 06:21:15.320403 16304 heartbeater.cc:347] Connected to a master server at 127.3.137.126:33565
I20260809 06:21:15.320708 16304 heartbeater.cc:464] Registering TS with master...
I20260809 06:21:15.321405 16304 heartbeater.cc:511] Master 127.3.137.126:33565 requested a full tablet report, sending...
I20260809 06:21:15.322762 15791 ts_manager.cc:194] Registered new tserver with Master: 71bb8e45f7e343edb80b4057eedd50f2 (127.3.137.67:37319)
I20260809 06:21:15.323864 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:47379
I20260809 06:21:15.325158 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:21:15.343479 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.330262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.3.137.126:33565)
0809 06:21:15.330408 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.330412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.331020 (+ 608us) server_negotiation.cc:407] Connection header received
0809 06:21:15.331354 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.331362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.331460 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.331571 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:15.333072 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.333816 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.335406 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.336197 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.336374 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.338476 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:15.338481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:15.338484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:15.338514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:15.341113 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.341679 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.341682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.341684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.341849 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.342212 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.342217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.342220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.342551 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:15.342747 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.343082 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.343251 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:21:15.347463 15791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:21:15.354828 15791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:21:15.384680 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.367796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.3.137.66:46701)
0809 06:21:15.368381 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.368391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.376209 (+ 7818us) server_negotiation.cc:407] Connection header received
0809 06:21:15.376565 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.376577 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.376887 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.378099 (+ 1212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:21:15.378363 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.380065 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.382367 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383393 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.383542 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.383780 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.384140 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.384334 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260809 06:21:15.384871 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.367642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:46701 (local address 127.0.0.1:60032)
0809 06:21:15.376013 (+ 8371us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.376051 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.376290 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.377177 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.377188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.377245 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:21:15.377846 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.377863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.380234 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.380241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.382192 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.382204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383603 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.383610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.383887 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.383946 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.384078 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.384336 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8128,"mutex_wait_us":5,"thread_start_us":1761,"threads_started":1}
I20260809 06:21:15.388074 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.369346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:37319 (local address 127.0.0.1:50986)
0809 06:21:15.377395 (+ 8049us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.377412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.377583 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.380165 (+ 2582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.380171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.380204 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:21:15.380827 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.380840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383379 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.383383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.385966 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.385980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.387302 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.387308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.387460 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.387491 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.387583 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.387702 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7901,"mutex_wait_us":53,"thread_start_us":7300,"threads_started":1}
I20260809 06:21:15.388181 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.369802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.3.137.67:37319)
0809 06:21:15.379516 (+ 9714us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.379526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.379560 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:21:15.379731 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.379739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.379968 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.381275 (+ 1307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:21:15.381568 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383210 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.386121 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.387108 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.387244 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.387456 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.387662 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.387827 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9540,"thread_start_us":9381,"threads_started":1}
I20260809 06:21:15.388898 16096 tablet_service.cc:1467] Processing CreateTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 (DEFAULT_TABLE table=test-table [id=e27bfa5adb7d469d81376cefebf14e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:21:15.389755 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.368865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52830)
0809 06:21:15.376904 (+ 8039us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.376922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.377026 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.380444 (+ 3418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.380449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.380478 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:21:15.381158 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.381170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383442 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.383448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.386985 (+ 3537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.387000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.388927 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.388934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.389119 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.389150 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.389240 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.389364 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7917,"mutex_wait_us":127,"thread_start_us":6767,"threads_started":1}
I20260809 06:21:15.390610 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.369090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.3.137.65:38803)
0809 06:21:15.379874 (+ 10784us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.379882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.379908 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:21:15.379997 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.380006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.380210 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.381354 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:21:15.381598 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.383245 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.387159 (+ 3914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.388692 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.388837 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.389081 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.390091 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.390250 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10609,"thread_start_us":10470,"threads_started":1}
I20260809 06:21:15.391075 16096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68cd05251994a74a6dc3acf939ae3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:21:15.391850 16234 tablet_service.cc:1467] Processing CreateTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 (DEFAULT_TABLE table=test-table [id=e27bfa5adb7d469d81376cefebf14e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:21:15.392966 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68cd05251994a74a6dc3acf939ae3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:21:15.394372 15947 tablet_service.cc:1467] Processing CreateTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 (DEFAULT_TABLE table=test-table [id=e27bfa5adb7d469d81376cefebf14e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:21:15.395896 15947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e68cd05251994a74a6dc3acf939ae3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:21:15.404575 16324 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Bootstrap starting.
I20260809 06:21:15.407721 16325 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Bootstrap starting.
I20260809 06:21:15.408424 16324 tablet_bootstrap.cc:654] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:21:15.409759 16324 log.cc:824] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:21:15.410250 16326 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Bootstrap starting.
I20260809 06:21:15.411649 16325 tablet_bootstrap.cc:654] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Neither blocks nor log segments found. Creating new log.
I20260809 06:21:15.413334 16325 log.cc:824] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Log is configured to *not* fsync() on all Append() calls
I20260809 06:21:15.413754 16326 tablet_bootstrap.cc:654] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Neither blocks nor log segments found. Creating new log.
I20260809 06:21:15.415222 16326 log.cc:824] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Log is configured to *not* fsync() on all Append() calls
I20260809 06:21:15.416812 16324 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: No bootstrap required, opened a new log
I20260809 06:21:15.416874 16325 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: No bootstrap required, opened a new log
I20260809 06:21:15.417100 16324 ts_tablet_manager.cc:1397] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260809 06:21:15.417131 16325 ts_tablet_manager.cc:1397] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 06:21:15.418417 16326 tablet_bootstrap.cc:492] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: No bootstrap required, opened a new log
I20260809 06:21:15.418677 16326 ts_tablet_manager.cc:1397] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:21:15.424511 16325 raft_consensus.cc:348] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.424873 16325 raft_consensus.cc:374] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:21:15.425004 16325 raft_consensus.cc:729] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f623160f9e24ec8a43d84551cde0ad1, State: Initialized, Role: FOLLOWER
I20260809 06:21:15.425573 16325 consensus_queue.cc:260] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.426661 16163 heartbeater.cc:503] Master 127.3.137.126:33565 was elected leader, sending a full tablet report...
I20260809 06:21:15.427400 16325 ts_tablet_manager.cc:1428] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260809 06:21:15.427896 16324 raft_consensus.cc:348] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.428311 16324 raft_consensus.cc:374] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:21:15.428416 16324 raft_consensus.cc:729] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71bb8e45f7e343edb80b4057eedd50f2, State: Initialized, Role: FOLLOWER
I20260809 06:21:15.429036 16324 consensus_queue.cc:260] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.429693 16326 raft_consensus.cc:348] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.430238 16326 raft_consensus.cc:374] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:21:15.430233 16304 heartbeater.cc:503] Master 127.3.137.126:33565 was elected leader, sending a full tablet report...
I20260809 06:21:15.430497 16326 raft_consensus.cc:729] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243f81f49ff64baf832f9291afaeaadd, State: Initialized, Role: FOLLOWER
I20260809 06:21:15.430825 16324 ts_tablet_manager.cc:1428] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260809 06:21:15.431227 16326 consensus_queue.cc:260] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.432941 16326 ts_tablet_manager.cc:1428] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
W20260809 06:21:15.435225 16164 tablet.cc:2369] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:21:15.503468 16017 tablet.cc:2369] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:21:15.549767 16305 tablet.cc:2369] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:21:15.789712 16330 raft_consensus.cc:482] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:21:15.790081 16330 raft_consensus.cc:504] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.791822 16330 leader_election.cc:288] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 243f81f49ff64baf832f9291afaeaadd (127.3.137.65:38803), 71bb8e45f7e343edb80b4057eedd50f2 (127.3.137.67:37319)
I20260809 06:21:15.801265 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.792435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:49573 (local address 127.3.137.67:37319)
0809 06:21:15.792662 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.792668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.793188 (+ 520us) server_negotiation.cc:407] Connection header received
0809 06:21:15.793362 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.793368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.793450 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.793556 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:21:15.794369 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.795566 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.798640 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.799995 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.800170 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.800244 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.800862 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.801006 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 06:21:15.801339 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.792342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:59837 (local address 127.3.137.65:38803)
0809 06:21:15.792653 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.792663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.793117 (+ 454us) server_negotiation.cc:407] Connection header received
0809 06:21:15.793362 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.793368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.793455 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.793556 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:21:15.794369 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.795566 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.798633 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.799971 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.800166 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.800249 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.800968 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.801097 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 06:21:15.801517 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.792208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:37319 (local address 127.3.137.66:49573)
0809 06:21:15.793096 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.793117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.793212 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.793596 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.793601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.793629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:21:15.794116 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.794137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.795725 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.795729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.798440 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.798448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.800260 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.800265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.800577 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.800654 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.800750 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.800961 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":359,"thread_start_us":468,"threads_started":1}
I20260809 06:21:15.801525 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.792145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.3.137.66:59837)
0809 06:21:15.792985 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:15.793006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:21:15.793181 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:21:15.793596 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:21:15.793601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:21:15.793656 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:21:15.794195 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.794208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.795723 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.795729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.798432 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:21:15.798448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.800257 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.800263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:21:15.800573 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.800649 (+ 76us) client_negotiation.cc:769] Sending connection context
0809 06:21:15.800750 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 06:21:15.800962 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":276,"spinlock_wait_cycles":28928,"thread_start_us":677,"threads_started":1}
I20260809 06:21:15.802708 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68cd05251994a74a6dc3acf939ae3e6" candidate_uuid: "5f623160f9e24ec8a43d84551cde0ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bb8e45f7e343edb80b4057eedd50f2" is_pre_election: true
I20260809 06:21:15.802680 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68cd05251994a74a6dc3acf939ae3e6" candidate_uuid: "5f623160f9e24ec8a43d84551cde0ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243f81f49ff64baf832f9291afaeaadd" is_pre_election: true
I20260809 06:21:15.803216 16257 raft_consensus.cc:2454] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f623160f9e24ec8a43d84551cde0ad1 in term 0.
I20260809 06:21:15.803216 15969 raft_consensus.cc:2454] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f623160f9e24ec8a43d84551cde0ad1 in term 0.
I20260809 06:21:15.803853 16049 leader_election.cc:302] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 243f81f49ff64baf832f9291afaeaadd, 5f623160f9e24ec8a43d84551cde0ad1, 71bb8e45f7e343edb80b4057eedd50f2; no voters:
I20260809 06:21:15.804289 16330 raft_consensus.cc:2781] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:21:15.804445 16330 raft_consensus.cc:482] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:21:15.804564 16330 raft_consensus.cc:3037] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:21:15.807159 16330 raft_consensus.cc:504] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.807871 16330 leader_election.cc:288] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 243f81f49ff64baf832f9291afaeaadd (127.3.137.65:38803), 71bb8e45f7e343edb80b4057eedd50f2 (127.3.137.67:37319)
I20260809 06:21:15.808369 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68cd05251994a74a6dc3acf939ae3e6" candidate_uuid: "5f623160f9e24ec8a43d84551cde0ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243f81f49ff64baf832f9291afaeaadd"
I20260809 06:21:15.808445 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e68cd05251994a74a6dc3acf939ae3e6" candidate_uuid: "5f623160f9e24ec8a43d84551cde0ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bb8e45f7e343edb80b4057eedd50f2"
I20260809 06:21:15.808599 15969 raft_consensus.cc:3037] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:21:15.808632 16257 raft_consensus.cc:3037] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:21:15.812033 15969 raft_consensus.cc:2454] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f623160f9e24ec8a43d84551cde0ad1 in term 1.
I20260809 06:21:15.812033 16257 raft_consensus.cc:2454] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f623160f9e24ec8a43d84551cde0ad1 in term 1.
I20260809 06:21:15.812456 16051 leader_election.cc:302] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243f81f49ff64baf832f9291afaeaadd, 5f623160f9e24ec8a43d84551cde0ad1; no voters:
I20260809 06:21:15.812788 16330 raft_consensus.cc:2781] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:21:15.813232 16330 raft_consensus.cc:686] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 5f623160f9e24ec8a43d84551cde0ad1, State: Running, Role: LEADER
I20260809 06:21:15.813717 16330 consensus_queue.cc:237] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } }
I20260809 06:21:15.818277 15788 catalog_manager.cc:5495] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f623160f9e24ec8a43d84551cde0ad1 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "5f623160f9e24ec8a43d84551cde0ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f623160f9e24ec8a43d84551cde0ad1" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 46701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 } health_report { overall_health: UNKNOWN } } }
I20260809 06:21:15.837383 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.832307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.3.137.66:46701)
0809 06:21:15.832516 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.832521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.832538 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:21:15.832697 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.832701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.832785 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.832903 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:21:15.833245 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.833926 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.835378 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.835771 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.835863 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.836301 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:21:15.836790 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:21:15.836896 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.836999 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.837060 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:21:15.845620 15969 raft_consensus.cc:1261] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 1 FOLLOWER]: Refusing update from remote peer 5f623160f9e24ec8a43d84551cde0ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:21:15.845623 16257 raft_consensus.cc:1261] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 1 FOLLOWER]: Refusing update from remote peer 5f623160f9e24ec8a43d84551cde0ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:21:15.846719 16330 consensus_queue.cc:1035] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71bb8e45f7e343edb80b4057eedd50f2" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:21:15.847163 16336 consensus_queue.cc:1035] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243f81f49ff64baf832f9291afaeaadd" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 38803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:21:15.863494 16341 mvcc.cc:205] Tried to move back new op lower bound from 7316506525052231680 to 7316506524936527872. Current Snapshot: MvccSnapshot[applied={T|T < 7316506525052231680}]
I20260809 06:21:15.872287 15790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:21:15.872583 15790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:21:15.876195 15790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d268dbb79d445e383ead3c6e7891074: Sending DeleteTablet for 3 replicas of tablet e68cd05251994a74a6dc3acf939ae3e6
I20260809 06:21:15.877310 16096 tablet_service.cc:1514] Processing DeleteTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:21:15 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:60032
I20260809 06:21:15.877337 15947 tablet_service.cc:1514] Processing DeleteTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:21:15 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:52830
I20260809 06:21:15.878001 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet e68cd05251994a74a6dc3acf939ae3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:21:15 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:50986
I20260809 06:21:15.878288 16348 tablet_replica.cc:330] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: stopping tablet replica
I20260809 06:21:15.878731 16348 raft_consensus.cc:2229] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:21:15.879609 16348 raft_consensus.cc:2258] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:21:15.881834 16348 ts_tablet_manager.cc:1905] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:21:15.886339 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 15875
I20260809 06:21:15.891999 16348 ts_tablet_manager.cc:1918] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:21:15.892236 16348 log.cc:1196] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/e68cd05251994a74a6dc3acf939ae3e6
I20260809 06:21:15.892923 16348 ts_tablet_manager.cc:1939] T e68cd05251994a74a6dc3acf939ae3e6 P 5f623160f9e24ec8a43d84551cde0ad1: Deleting consensus metadata
I20260809 06:21:15.895028 15775 catalog_manager.cc:4841] TS 5f623160f9e24ec8a43d84551cde0ad1 (127.3.137.66:46701): tablet e68cd05251994a74a6dc3acf939ae3e6 (table test-table [id=e27bfa5adb7d469d81376cefebf14e1d]) successfully deleted
I20260809 06:21:15.906324 16351 tablet_replica.cc:330] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: stopping tablet replica
I20260809 06:21:15.906610 16350 tablet_replica.cc:330] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: stopping tablet replica
I20260809 06:21:15.906905 16351 raft_consensus.cc:2229] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:21:15.907223 16350 raft_consensus.cc:2229] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:21:15.907564 16351 raft_consensus.cc:2258] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:21:15.907758 16350 raft_consensus.cc:2258] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:21:15.909762 16351 ts_tablet_manager.cc:1905] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:21:15.912221 16350 ts_tablet_manager.cc:1905] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:21:15 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:21:15.916350 16351 ts_tablet_manager.cc:1918] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:21:15.916482 16351 log.cc:1196] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/e68cd05251994a74a6dc3acf939ae3e6
I20260809 06:21:15.916941 16351 ts_tablet_manager.cc:1939] T e68cd05251994a74a6dc3acf939ae3e6 P 71bb8e45f7e343edb80b4057eedd50f2: Deleting consensus metadata
I20260809 06:21:15.918676 15774 catalog_manager.cc:4841] TS 71bb8e45f7e343edb80b4057eedd50f2 (127.3.137.67:37319): tablet e68cd05251994a74a6dc3acf939ae3e6 (table test-table [id=e27bfa5adb7d469d81376cefebf14e1d]) successfully deleted
I20260809 06:21:15.924499 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:15.887970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.3.137.65:38803)
0809 06:21:15.888172 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:15.888181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:15.902860 (+ 14679us) server_negotiation.cc:407] Connection header received
0809 06:21:15.903250 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:15.903260 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:15.903369 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:15.903462 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:15.906490 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.907796 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.909957 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:15.910590 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:15.910691 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:15.916530 (+ 5839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:15.916544 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:15.916563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:15.916616 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:15.920768 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.921706 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.921725 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.921729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.921903 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:15.922397 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:15.922402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:15.922406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:15.922863 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:15.923179 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:15.923794 (+ 615us) server_negotiation.cc:299] Negotiation successful
0809 06:21:15.924083 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:21:15.926026 16011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:21:16.043608 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 15875
I20260809 06:21:16.044026 16350 ts_tablet_manager.cc:1918] T e68cd05251994a74a6dc3acf939ae3e6 P 243f81f49ff64baf832f9291afaeaadd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260809 06:21:16.070727 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:38803 (error 108)
I20260809 06:21:16.071023 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 16024
I20260809 06:21:16.073583 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.072448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52848)
0809 06:21:16.073292 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.073404 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":623,"threads_started":1}
W20260809 06:21:16.073865 15776 catalog_manager.cc:4584] TS 243f81f49ff64baf832f9291afaeaadd (127.3.137.65:38803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e68cd05251994a74a6dc3acf939ae3e6: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Aug 09 06:21:16 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:21:16.089972 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.072147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.3.137.66:46701)
0809 06:21:16.072342 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:16.072348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:16.072369 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:21:16.072488 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:16.072495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:16.072630 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:16.072768 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:16.074706 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:16.075646 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:16.077681 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:16.078107 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:16.078207 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:16.083760 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:16.083775 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:16.083792 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:16.083830 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:16.087055 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:16.087734 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:16.087751 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:16.087755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:16.087900 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:16.088311 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:16.088316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:16.088321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:16.088694 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:16.089059 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:16.089342 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 06:21:16.089596 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:21:16.090710 16158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:21:16.127566 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.127104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52854)
0809 06:21:16.127324 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.127426 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:21:16.128703 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.128309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52862)
0809 06:21:16.128500 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.128572 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 06:21:16.189505 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.188959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52872)
0809 06:21:16.189186 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.189345 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 06:21:16.190626 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.190257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52882)
0809 06:21:16.190418 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.190491 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 06:21:16.208811 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 16024
I20260809 06:21:16.229779 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 16170
Aug 09 06:21:16 dist-test-slave-2w3w krb5kdc[15288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:21:16.245723 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.230579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.3.137.67:37319)
0809 06:21:16.230835 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:21:16.230841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:21:16.230863 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:21:16.231104 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:21:16.231110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:21:16.231189 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:21:16.231304 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:21:16.232840 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:16.233571 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:16.234908 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:21:16.235426 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:21:16.235579 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:21:16.240094 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:21:16.240119 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:21:16.240135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:21:16.240175 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:21:16.242808 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:16.243543 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:16.243553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:16.243557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:16.243701 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:21:16.244136 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:21:16.244141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:21:16.244145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:21:16.244526 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:21:16.244834 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 06:21:16.245134 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 06:21:16.245358 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:21:16.246387 16299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:21:16.271111 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.270627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52896)
0809 06:21:16.270825 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.270905 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 06:21:16.272105 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.271726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52906)
0809 06:21:16.271918 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.271983 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 06:21:16.338088 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 16170
I20260809 06:21:16.356690 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 15757
I20260809 06:21:16.357954 15824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:21:16.471944 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:21:16.471242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:38803 (local address 127.0.0.1:52914)
0809 06:21:16.471447 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 06:21:16.471617 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 06:21:16.472138 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 15757
I20260809 06:21:16.500254 3621 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 06:21:16.844913 3621 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 06:21:16.845232 3621 mini_ranger.cc:66] Stopping Ranger...
I20260809 06:21:17.190335 3621 mini_ranger.cc:68] Stopped Ranger
2026-08-09 06:21:17.190 UTC [15320] LOG: received smart shutdown request
2026-08-09 06:21:17.194 UTC [15320] LOG: background worker "logical replication launcher" (PID 15330) exited with exit code 1
2026-08-09 06:21:17.195 UTC [15325] LOG: shutting down
2026-08-09 06:21:17.343 UTC [15320] LOG: database system is shut down
2026-08-09T06:21:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50917 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:21:17 dist-test-slave-2w3w krb5kdc[16367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:21:17 dist-test-slave-2w3w krb5kdc[16367](info): set up 2 sockets
Aug 09 06:21:17 dist-test-slave-2w3w krb5kdc[16367](info): commencing operation
krb5kdc: starting...
W20260809 06:21:18.635926 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:21:18 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:21:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangeradmin_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangeradmin_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangerlookup_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangerlookup_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
I20260809 06:21:19.024247 3621 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Z6xHV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/postgres -l logfile start
2026-08-09 06:21:21.425 UTC [16398] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 06:21:21.425 UTC [16398] LOG: listening on IPv4 address "127.3.137.84", port 44965
2026-08-09 06:21:21.428 UTC [16398] LOG: listening on Unix socket "/tmp/.s.PGSQL.44965"
2026-08-09 06:21:21.452 UTC [16401] LOG: database system was shut down at 2026-08-09 06:21:20 UTC
2026-08-09 06:21:21.456 UTC [16398] LOG: database system is ready to accept connections
I20260809 06:21:22.480160 3621 mini_postgres.cc:96] Postgres bound to 44965
2026-08-09 06:21:22.503 UTC [16409] FATAL: database "slave" does not exist
127.3.137.84:44965 - accepting connections
I20260809 06:21:22.504472 3621 mini_ranger.cc:161] Starting Ranger...
I20260809 06:21:22.535476 3621 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 06:21:22.971186 3621 mini_ranger.cc:87] Created ranger Postgres database
I20260809 06:21:22.971312 3621 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin
2026-08-09 06:21:23,280 [I] DB FLAVOR :POSTGRES
2026-08-09 06:21:23,280 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:21:23,280 [I] Checking connection..
2026-08-09 06:21:23,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 06:21:23,546 [I] Checking connection passed.
2026-08-09 06:21:23,546 [I] --------- Verifying version history table ---------
2026-08-09 06:21:23,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:21:23,802 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:21:23,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:21:24,069 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:21:24,069 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 06:21:24,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 06:21:24.304 UTC [16479] WARNING: there is no transaction in progress
2026-08-09 06:21:24,312 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 06:21:24,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:21:24,565 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 06:21:24,565 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:21:24,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 06:21:24,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 06:21:25,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-09 06:21:25,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:21:25,589 [I] Table x_portal_user does not exist in database ranger
2026-08-09 06:21:25,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:21:25,850 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 06:21:25,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:21:26,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:21:26,329 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 06:21:26,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 06:21:27.270 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.286 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.301 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.315 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.331 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.347 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.399 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.405 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.419 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.429 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.439 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.451 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.460 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.465 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.471 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:27.476 UTC [16595] WARNING: there is no transaction in progress
2026-08-09 06:21:28,074 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 06:21:28,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 06:21:28,305 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 06:21:28,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:21:28,564 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 06:21:28,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:21:28,804 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 06:21:28,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:21:29,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:21:29,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-09 06:21:29,558 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 06:21:29,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 06:21:29,811 [I] DB_PATCHES have already been applied
I20260809 06:21:29.817893 3621 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Z6xHV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 06:21:29.817983 3621 mini_ranger.cc:193] Using host: 127.3.137.84
I20260809 06:21:29.839071 3621 mini_ranger.cc:227] Ranger admin URL: http://127.3.137.84:41787
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 6:21:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 06:21:31.027063 3621 mini_ranger.cc:160] Time spent starting Ranger: real 8.523s user 0.003s sys 0.054s
Aug 09, 2026 6:21:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 6:21:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 09, 2026 6:21:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 6:21:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 6:21:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 06:21:52.537377 3621 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.137.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.137.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangerkms_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.137.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/rangerkms_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.137.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.137.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.137.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/HTTP_127.3.137.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 06:21:52 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256512, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 06:21:52.744315 3621 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 06:21:52.768980 3621 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 06:21:53.565660 3621 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 06:21:53.565759 3621 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms
I20260809 06:21:53.565791 3621 mini_ranger_kms.cc:227] Using postgres at 127.3.137.84:44965
2026-08-09 06:21:53,959 [I] DB FLAVOR :POSTGRES
2026-08-09 06:21:53,960 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:21:53,960 [I] Checking connection
2026-08-09 06:21:53,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 06:21:54,224 [I] connection success
2026-08-09 06:21:54,225 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 06:21:54,225 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 06:21:54,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 06:21:54,467 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 06:21:54,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:21:54,468 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 06:21:54,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.137.84:44965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 06:21:54,719 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 06:21:55.173481 3621 mini_ranger_kms.cc:310] Created kms service
I20260809 06:21:55.380393 3621 mini_ranger_kms.cc:326] Created kudu user
I20260809 06:21:55.430785 3621 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 06:21:55 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256512, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:21:55.863279 3621 mini_ranger_kms.cc:384] Added ranger policy
I20260809 06:21:55.863623 3621 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Z6xHV/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Z6xHV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 06:21:55.863696 3621 mini_ranger_kms.cc:240] Using host: 127.3.137.84
I20260809 06:21:55.885802 3621 mini_ranger_kms.cc:276] Ranger KMS PID: 16796
I20260809 06:21:55.885913 3621 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.137.84:41015
Aug 09, 2026 6:21:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:21:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:21:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 6:21:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 6:21:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 6:21:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 06:21:57 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786256517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:21:58.275024 3621 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.531s user 0.088s sys 0.339s
I20260809 06:21:58.275297 3621 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 06:21:58.275476 3621 mini_ranger_kms.cc:401] 127.3.137.84:41015/kms/v1/keys
Aug 09 06:21:58 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
Aug 09 06:21:58 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256517, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.137.84@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
Aug 09 06:22:01 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:22:01.072832 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46839
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:46839
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:01.407054 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:01.407327 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:01.407411 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:01.414523 16850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:01.414625 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:01.414671 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:01.414743 16850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:01.414793 16850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:01.424538 16850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.137.84:41015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:46839
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:01.425714 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:01.427078 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:01.439384 16855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:01.439468 16859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:01.439420 16857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:01.440286 16850 server_base.cc:1034] running on GCE node
I20260809 06:22:01.441087 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:01.442482 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:01.443707 16850 hybrid_clock.cc:648] HybridClock initialized: now 1786256521443667 us; error 59 us; skew 500 ppm
I20260809 06:22:01.444067 16850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:01 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:01.452354 16850 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:01.453961 16850 webserver.cc:460] Webserver started at http://127.3.137.126:35111/ using document root <none> and password file <none>
I20260809 06:22:01.454633 16850 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:01.454802 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:01.455191 16850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:22:01 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:01.505781 16850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "374a45eaef93464d8b42fabe9fda4ba9"
format_stamp: "Formatted at 2026-08-09 06:22:01 on dist-test-slave-2w3w"
server_key: "d7c933975eaca387eafb1484a9ec4693"
server_key_iv: "953d70014daa511b88db37d5d15666fe"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:01.506764 16850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "374a45eaef93464d8b42fabe9fda4ba9"
format_stamp: "Formatted at 2026-08-09 06:22:01 on dist-test-slave-2w3w"
server_key: "d7c933975eaca387eafb1484a9ec4693"
server_key_iv: "953d70014daa511b88db37d5d15666fe"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:01.512461 16850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:22:01.532218 16866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:01.533769 16850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:22:01.533967 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "374a45eaef93464d8b42fabe9fda4ba9"
format_stamp: "Formatted at 2026-08-09 06:22:01 on dist-test-slave-2w3w"
server_key: "d7c933975eaca387eafb1484a9ec4693"
server_key_iv: "953d70014daa511b88db37d5d15666fe"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:01.534186 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:01.557904 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:01.561635 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:01.561913 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:01.580322 16850 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:46839
I20260809 06:22:01.580355 16919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:46839 every 8 connection(s)
I20260809 06:22:01.581617 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:01.585284 16920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:01.589108 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 16850
I20260809 06:22:01.589340 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:22:01.590229 3621 external_mini_cluster.cc:1428] Setting key fde319bd748689adc0d13eae83c66cb9
I20260809 06:22:01.596784 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Bootstrap starting.
I20260809 06:22:01.599790 16920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:01.600777 16920 log.cc:824] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:01.603531 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: No bootstrap required, opened a new log
Aug 09 06:22:01 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:01.611557 16920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:01.611982 16920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:01.612115 16920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a45eaef93464d8b42fabe9fda4ba9, State: Initialized, Role: FOLLOWER
I20260809 06:22:01.612720 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:01.592448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.3.137.126:46839)
0809 06:22:01.593192 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:01.593203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:01.593241 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:22:01.595527 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:01.595549 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:01.595907 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:01.596271 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:01.597674 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:01.599046 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:01.600416 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:01.601140 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:01.601371 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:01.605486 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:01.605497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:01.605514 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:01.605548 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:01.608686 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:01.609364 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:01.609370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:01.609375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:01.609517 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:01.609927 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:01.609934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:01.609942 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:01.610467 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:01.610803 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:01.611225 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 06:22:01.611610 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":354,"threads_started":1}
I20260809 06:22:01.612758 16920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:01.613083 16920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:01.613243 16920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:01.613446 16920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:01.615906 16920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:01.616359 16920 leader_election.cc:302] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 374a45eaef93464d8b42fabe9fda4ba9; no voters:
I20260809 06:22:01.616895 16920 leader_election.cc:288] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:22:01.617026 16925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:01.617843 16925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 374a45eaef93464d8b42fabe9fda4ba9, State: Running, Role: LEADER
I20260809 06:22:01.618444 16925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:01.619349 16920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:01.621709 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 374a45eaef93464d8b42fabe9fda4ba9. Latest consensus state: current_term: 1 leader_uuid: "374a45eaef93464d8b42fabe9fda4ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } } }
I20260809 06:22:01.622169 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:01.622764 16926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "374a45eaef93464d8b42fabe9fda4ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } } }
I20260809 06:22:01.623175 16926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:01.628115 16934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:01.628628 16920 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:22:01.645633 16934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:01.657347 16934 catalog_manager.cc:1347] Generated new cluster ID: ece867be2ac24a1eafa693c4e400f860
I20260809 06:22:01.657480 16934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:01.685699 16934 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:22:01.687901 16934 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:01.697021 16934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Generated new TSK 0
I20260809 06:22:01.698133 16934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:22:02.410923 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:02 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 06:22:02.880528 16967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:02.880996 16967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:02.881191 16967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:02.893052 16967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:02.893196 16967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:02.893440 16967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:02.909265 16967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:02.911113 16967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:02.912753 16967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:02.929096 16977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:02.930716 16978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:02.935508 16967 server_base.cc:1034] running on GCE node
W20260809 06:22:02.935828 16980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:02.936697 16967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:02.937978 16967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:02.939267 16967 hybrid_clock.cc:648] HybridClock initialized: now 1786256522939220 us; error 38 us; skew 500 ppm
I20260809 06:22:02.939620 16967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:02 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:02.943910 16967 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:02.965857 16967 webserver.cc:460] Webserver started at http://127.3.137.65:33895/ using document root <none> and password file <none>
I20260809 06:22:02.966998 16967 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:02.967168 16967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:02.967566 16967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:22:02 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:02.989691 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "b8812cf5d6b6488bb8089b2280c4ecaf"
format_stamp: "Formatted at 2026-08-09 06:22:02 on dist-test-slave-2w3w"
server_key: "d7400bc6a7668b7c58bf545f1f1f75e0"
server_key_iv: "f2c867cdaf4a9f96d544a8ffc4421981"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:02.990792 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "b8812cf5d6b6488bb8089b2280c4ecaf"
format_stamp: "Formatted at 2026-08-09 06:22:02 on dist-test-slave-2w3w"
server_key: "d7400bc6a7668b7c58bf545f1f1f75e0"
server_key_iv: "f2c867cdaf4a9f96d544a8ffc4421981"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:02.997804 16967 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260809 06:22:03.021646 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:03.023679 16967 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.000s
I20260809 06:22:03.023874 16967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "b8812cf5d6b6488bb8089b2280c4ecaf"
format_stamp: "Formatted at 2026-08-09 06:22:02 on dist-test-slave-2w3w"
server_key: "d7400bc6a7668b7c58bf545f1f1f75e0"
server_key_iv: "f2c867cdaf4a9f96d544a8ffc4421981"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:03.024156 16967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:03.047915 16967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:03.053318 16967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:03.053648 16967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:03.055866 16967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:03.057529 16967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:03.057617 16967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:03.057736 16967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:03.057789 16967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:03.142373 16967 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:43187
I20260809 06:22:03.142401 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:43187 every 8 connection(s)
I20260809 06:22:03.143759 16967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:03.149966 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 16967
I20260809 06:22:03.150215 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:22:03.151247 3621 external_mini_cluster.cc:1428] Setting key fd6a21ec8d4ca15672957e7535355fca
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
I20260809 06:22:03.186314 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:03.146938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:46159 (local address 127.3.137.126:46839)
0809 06:22:03.155357 (+ 8419us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:03.155365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:03.156765 (+ 1400us) server_negotiation.cc:407] Connection header received
0809 06:22:03.162189 (+ 5424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:03.162195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:03.162308 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:03.162398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:03.164747 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.165948 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.168615 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.169228 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.169344 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:03.175526 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:03.175533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:03.175538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:03.175580 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:03.181640 (+ 6060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:03.182792 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:03.182797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:03.182801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:03.182987 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:03.183593 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:03.183599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:03.183602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:03.183983 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:03.184121 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:03.185664 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0809 06:22:03.185903 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8264,"thread_start_us":8105,"threads_started":1}
I20260809 06:22:03.187497 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:03.147639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.65:46159)
0809 06:22:03.156562 (+ 8923us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:03.156620 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:03.159344 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:03.163008 (+ 3664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:03.163022 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:03.163685 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:03.164532 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:03.164560 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.166166 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.166171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:03.168436 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:03.168452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.169701 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.169709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:03.169985 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:03.170812 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:03.170844 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:03.175232 (+ 4388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:03.181994 (+ 6762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:03.182003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:03.182022 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:03.182642 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:03.183153 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:03.183158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:03.183162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:03.183453 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:03.184697 (+ 1244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:03.184707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:03.185166 (+ 459us) client_negotiation.cc:769] Sending connection context
0809 06:22:03.185512 (+ 346us) client_negotiation.cc:240] Negotiation successful
0809 06:22:03.186099 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8585,"thread_start_us":8334,"threads_started":1}
I20260809 06:22:03.189577 17106 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:03.189916 17106 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
I20260809 06:22:03.190785 17106 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:03.193284 16883 ts_manager.cc:194] Registered new tserver with Master: b8812cf5d6b6488bb8089b2280c4ecaf (127.3.137.65:43187)
I20260809 06:22:03.195369 16883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:46159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:22:03.236558 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
W20260809 06:22:03.718247 17112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:03.718595 17112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:03.718777 17112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:03.726938 17112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:03.727150 17112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:03.727363 17112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:03.737993 17112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:03.739427 17112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:03.740896 17112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:03.776611 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:03.776827 17130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:03.777163 17120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:03.778039 17112 server_base.cc:1034] running on GCE node
I20260809 06:22:03.778748 17112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:03.779868 17112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:03.781123 17112 hybrid_clock.cc:648] HybridClock initialized: now 1786256523781041 us; error 78 us; skew 500 ppm
I20260809 06:22:03.781454 17112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:03.784655 17112 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:03.786134 17112 webserver.cc:460] Webserver started at http://127.3.137.66:37561/ using document root <none> and password file <none>
I20260809 06:22:03.786809 17112 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:03.787019 17112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:03.787340 17112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:03.804138 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "7a12f397821b420eb649c92fbe2dc236"
format_stamp: "Formatted at 2026-08-09 06:22:03 on dist-test-slave-2w3w"
server_key: "0210d0726835c01a5d757e70c73e875b"
server_key_iv: "a532c572777e5dd17255358b7962d824"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:03.805222 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "7a12f397821b420eb649c92fbe2dc236"
format_stamp: "Formatted at 2026-08-09 06:22:03 on dist-test-slave-2w3w"
server_key: "0210d0726835c01a5d757e70c73e875b"
server_key_iv: "a532c572777e5dd17255358b7962d824"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:03.812041 17112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260809 06:22:03.828888 17137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:03.830433 17112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:22:03.830721 17112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "7a12f397821b420eb649c92fbe2dc236"
format_stamp: "Formatted at 2026-08-09 06:22:03 on dist-test-slave-2w3w"
server_key: "0210d0726835c01a5d757e70c73e875b"
server_key_iv: "a532c572777e5dd17255358b7962d824"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:03.830950 17112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:03.848162 17112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:03.852214 17112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:03.852506 17112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:03.853420 17112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:03.854771 17112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:03.854871 17112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:03.855028 17112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:03.855104 17112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 06:22:03.899387 17112 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:35909
I20260809 06:22:03.899444 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:35909 every 8 connection(s)
I20260809 06:22:03.900683 17112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:03.909971 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17112
I20260809 06:22:03.910169 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:22:03.910765 3621 external_mini_cluster.cc:1428] Setting key 283afa58421fea30775f545aed14ad71
Aug 09 06:22:03 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:03.927137 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:03.903581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:42839 (local address 127.3.137.126:46839)
0809 06:22:03.904170 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:03.904177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:03.905348 (+ 1171us) server_negotiation.cc:407] Connection header received
0809 06:22:03.907844 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:03.907849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:03.907917 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:03.908014 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:03.909947 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.911093 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.913444 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.914168 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.914335 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:03.918852 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:03.918861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:03.918868 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:03.918913 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:03.922923 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:03.924051 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:03.924056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:03.924060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:03.924216 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:03.924854 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:03.924861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:03.924865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:03.925254 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:03.925446 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:03.926358 (+ 912us) server_negotiation.cc:299] Negotiation successful
0809 06:22:03.926612 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
I20260809 06:22:03.928597 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:03.904329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.66:42839)
0809 06:22:03.905165 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:03.905216 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:03.907546 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:03.908300 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:03.908312 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:03.908929 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:03.909737 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:03.909762 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.911282 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.911287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:03.913277 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:03.913293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:03.914442 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:03.914450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:03.914772 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:03.915617 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:03.915645 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:03.918579 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:03.923105 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:03.923114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:03.923132 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:03.923900 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:03.924403 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:03.924408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:03.924412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:03.924727 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:03.925442 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:03.925459 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:03.925910 (+ 451us) client_negotiation.cc:769] Sending connection context
0809 06:22:03.926233 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 06:22:03.926785 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":331,"threads_started":1}
I20260809 06:22:03.930557 17262 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:03.930899 17262 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:03.931877 17262 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:03.933692 16883 ts_manager.cc:194] Registered new tserver with Master: 7a12f397821b420eb649c92fbe2dc236 (127.3.137.66:35909)
I20260809 06:22:03.934984 16883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:42839
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:22:03.976079 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:04.199143 17106 heartbeater.cc:503] Master 127.3.137.126:46839 was elected leader, sending a full tablet report...
W20260809 06:22:04.341274 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:04.341562 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:04.341730 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:04.348899 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:04.349033 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:04.349206 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:04.358647 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:04.359889 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:04.361142 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:04.373974 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:04.374207 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:04.374413 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:04.375247 17268 server_base.cc:1034] running on GCE node
I20260809 06:22:04.375842 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:04.376720 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:04.377933 17268 hybrid_clock.cc:648] HybridClock initialized: now 1786256524377895 us; error 29 us; skew 500 ppm
I20260809 06:22:04.378239 17268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:04.381388 17268 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:04.382763 17268 webserver.cc:460] Webserver started at http://127.3.137.67:36531/ using document root <none> and password file <none>
I20260809 06:22:04.383446 17268 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:04.383579 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:04.383910 17268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:04.399235 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "9820e870d0984855ba6604994be21568"
format_stamp: "Formatted at 2026-08-09 06:22:04 on dist-test-slave-2w3w"
server_key: "bcc4a3017299ce19528577d3a57644be"
server_key_iv: "2ff1e3642ffef2251bdc1b281423e947"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:04.400059 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "9820e870d0984855ba6604994be21568"
format_stamp: "Formatted at 2026-08-09 06:22:04 on dist-test-slave-2w3w"
server_key: "bcc4a3017299ce19528577d3a57644be"
server_key_iv: "2ff1e3642ffef2251bdc1b281423e947"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:04.405367 17268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:22:04.422698 17284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:04.424503 17268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:22:04.424724 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "9820e870d0984855ba6604994be21568"
format_stamp: "Formatted at 2026-08-09 06:22:04 on dist-test-slave-2w3w"
server_key: "bcc4a3017299ce19528577d3a57644be"
server_key_iv: "2ff1e3642ffef2251bdc1b281423e947"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:04.424994 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:04.461520 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:04.465164 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:04.465422 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:04.466300 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:04.467746 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:04.467820 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:04.467909 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:04.467953 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:04.501332 17268 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:45183
I20260809 06:22:04.501366 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:45183 every 8 connection(s)
I20260809 06:22:04.502622 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:04.508415 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17268
I20260809 06:22:04.508608 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:22:04.509073 3621 external_mini_cluster.cc:1428] Setting key 96ee892b58b3e43378af5df98f5c6e94
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:04.522333 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:04.505200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:58475 (local address 127.3.137.126:46839)
0809 06:22:04.505667 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:04.505672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:04.506651 (+ 979us) server_negotiation.cc:407] Connection header received
0809 06:22:04.508484 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:04.508490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:04.508582 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:04.508717 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:04.510183 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.510922 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.512565 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.512981 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.513113 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:04.516480 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:04.516485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:04.516488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:04.516526 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:04.519338 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.520045 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.520051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.520055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.520237 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.520649 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.520654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.520658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.520947 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:04.521112 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:04.521739 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 06:22:04.521966 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260809 06:22:04.523501 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:04.505795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.67:58475)
0809 06:22:04.506494 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:04.506533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:04.508209 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:04.508946 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:04.508953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:04.509450 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:04.510012 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:04.510029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.511113 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.511118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:04.512415 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:04.512427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.513215 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.513222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:04.513469 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:04.514090 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:04.514112 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:04.516245 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:04.519472 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:04.519480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:04.519496 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:04.519907 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:04.520348 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:04.520353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:04.520356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:04.520534 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:04.521088 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:04.521095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:04.521399 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 06:22:04.521663 (+ 264us) client_negotiation.cc:240] Negotiation successful
0809 06:22:04.522139 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":280,"threads_started":1}
I20260809 06:22:04.524956 17403 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:04.525274 17403 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:04.525970 17403 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:04.527443 16883 ts_manager.cc:194] Registered new tserver with Master: 9820e870d0984855ba6604994be21568 (127.3.137.67:45183)
I20260809 06:22:04.528215 16883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:58475
I20260809 06:22:04.536857 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:22:04.538697 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 16967
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:22:04.555210 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:04.539554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.3.137.65:43187)
0809 06:22:04.540188 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:04.540194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:04.540219 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:22:04.540273 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:04.540281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:04.540466 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:04.540595 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:04.542112 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.543324 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.544889 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.545380 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.545479 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:04.549814 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:04.549825 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:04.549838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:04.549869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:04.552790 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.553363 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.553369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.553372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.553511 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.553847 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.553850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.553852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.554152 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:04.554402 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:04.554649 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 06:22:04.554863 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":345,"threads_started":1}
I20260809 06:22:04.556382 17101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:04.646718 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 16967
I20260809 06:22:04.665758 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17112
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:22:04.683370 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:04.666692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.3.137.66:35909)
0809 06:22:04.667283 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:04.667290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:04.667314 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:22:04.667380 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:04.667386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:04.667634 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:04.667782 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:04.669355 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.671048 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.672456 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.673007 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.673106 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:04.677542 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:04.677553 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:04.677568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:04.677602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:04.680693 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.681364 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.681371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.681374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.681520 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.681856 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.681859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.681861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.682186 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:04.682469 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:04.682668 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 06:22:04.682950 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":329,"threads_started":1}
I20260809 06:22:04.684577 17257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:04.767483 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17112
I20260809 06:22:04.785485 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17268
Aug 09 06:22:04 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:22:04.802556 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:04.786489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.3.137.67:45183)
0809 06:22:04.787238 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:04.787245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:04.787277 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:22:04.787339 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:04.787345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:04.787554 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:04.787725 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:04.789319 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.790380 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.791811 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:04.792357 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:04.792494 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:04.796904 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:04.796916 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:04.796930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:04.796968 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:04.799749 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.800430 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.800437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.800439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.800570 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:04.800955 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:04.800962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:04.800966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:04.801303 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:04.801648 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:04.801927 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 06:22:04.802203 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260809 06:22:04.804175 17398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:04.887656 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17268
I20260809 06:22:04.905017 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 16850
I20260809 06:22:04.906384 16915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:05.014516 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 16850
I20260809 06:22:05.041239 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46839
--webserver_interface=127.3.137.126
--webserver_port=35111
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:46839
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:05.376060 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:05.376418 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:05.376605 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:05.383872 17417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:05.384027 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:05.384135 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:05.384243 17417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:05.384356 17417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:05.394330 17417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:46839
--ranger_config_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=35111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:05.395694 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:05.397059 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:05.411494 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:05.411432 17426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:05.411532 17424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:05.412284 17417 server_base.cc:1034] running on GCE node
I20260809 06:22:05.413321 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:05.415388 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:05.416656 17417 hybrid_clock.cc:648] HybridClock initialized: now 1786256525416591 us; error 54 us; skew 500 ppm
I20260809 06:22:05.417104 17417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:05 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:05.421392 17417 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:05.422886 17417 webserver.cc:460] Webserver started at http://127.3.137.126:35111/ using document root <none> and password file <none>
I20260809 06:22:05.423619 17417 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:05.423727 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:22:05 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:05.467839 17417 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:22:05.471642 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:05.473201 17417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:22:05.473382 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "374a45eaef93464d8b42fabe9fda4ba9"
format_stamp: "Formatted at 2026-08-09 06:22:01 on dist-test-slave-2w3w"
server_key: "d7c933975eaca387eafb1484a9ec4693"
server_key_iv: "953d70014daa511b88db37d5d15666fe"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:05.474350 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:05.496553 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:05.500284 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:05.500535 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:05.519485 17417 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:46839
I20260809 06:22:05.519505 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:46839 every 8 connection(s)
I20260809 06:22:05.520960 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:05.528421 17488 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:22:05.531739 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17417
I20260809 06:22:05.532395 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Bootstrap starting.
I20260809 06:22:05.532951 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:43187
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:46839
--webserver_port=33895
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:05.558545 17488 log.cc:824] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:05.574568 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:22:05.575340 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Bootstrap complete.
I20260809 06:22:05.588228 17488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:05.589107 17488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a45eaef93464d8b42fabe9fda4ba9, State: Initialized, Role: FOLLOWER
I20260809 06:22:05.589743 17488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:05.589979 17488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:05.590101 17488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:05.590292 17488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:22:05.594192 17488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:05.594734 17488 leader_election.cc:302] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 374a45eaef93464d8b42fabe9fda4ba9; no voters:
I20260809 06:22:05.595432 17488 leader_election.cc:288] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:22:05.595578 17495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:22:05.596772 17495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [term 2 LEADER]: Becoming Leader. State: Replica: 374a45eaef93464d8b42fabe9fda4ba9, State: Running, Role: LEADER
I20260809 06:22:05.597404 17495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } }
I20260809 06:22:05.597786 17488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:05.600342 17497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 374a45eaef93464d8b42fabe9fda4ba9. Latest consensus state: current_term: 2 leader_uuid: "374a45eaef93464d8b42fabe9fda4ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } } }
I20260809 06:22:05.600919 17497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:05.601495 17496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "374a45eaef93464d8b42fabe9fda4ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a45eaef93464d8b42fabe9fda4ba9" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46839 } } }
I20260809 06:22:05.601854 17496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:05.609977 17501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:05.616081 17501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:05.616900 17488 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:22:05.617511 17501 catalog_manager.cc:1259] Loaded cluster ID: ece867be2ac24a1eafa693c4e400f860
I20260809 06:22:05.617647 17501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:05.621532 17501 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:05.623001 17501 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Loaded TSK: 0
I20260809 06:22:05.624863 17501 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:22:06.014493 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:06.015033 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:06.015237 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:06.026974 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:06.027158 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:06.027383 17490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:06.043365 17490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:43187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=33895
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:06.045288 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:06.047103 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:06.065586 17531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:06.068873 17532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:06.076314 17490 server_base.cc:1034] running on GCE node
W20260809 06:22:06.083242 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:06.084072 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:06.085246 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:06.086550 17490 hybrid_clock.cc:648] HybridClock initialized: now 1786256526086475 us; error 68 us; skew 500 ppm
I20260809 06:22:06.087028 17490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:06.090744 17490 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:06.092630 17490 webserver.cc:460] Webserver started at http://127.3.137.65:33895/ using document root <none> and password file <none>
I20260809 06:22:06.093392 17490 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:06.093557 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:06.128813 17490 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 06:22:06.131821 17543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.133345 17490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:06.133538 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "b8812cf5d6b6488bb8089b2280c4ecaf"
format_stamp: "Formatted at 2026-08-09 06:22:02 on dist-test-slave-2w3w"
server_key: "d7400bc6a7668b7c58bf545f1f1f75e0"
server_key_iv: "f2c867cdaf4a9f96d544a8ffc4421981"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:06.134688 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:06.173027 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:06.178468 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:06.178787 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:06.180122 17490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:06.182013 17490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:06.182139 17490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.182257 17490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:06.182346 17490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.251176 17490 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:43187
I20260809 06:22:06.251258 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:43187 every 8 connection(s)
I20260809 06:22:06.252877 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:06.253760 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17490
I20260809 06:22:06.254647 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:35909
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:46839
--webserver_port=37561
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:06.319235 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:06.256408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.65:44213)
0809 06:22:06.278400 (+ 21992us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:06.278591 (+ 191us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:06.281551 (+ 2960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:06.283454 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:06.283467 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:06.284168 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:06.284986 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:06.285016 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.287417 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.287423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:06.289486 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:06.289503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.291566 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.291577 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:06.291916 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:06.292803 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:06.292839 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:06.306786 (+ 13947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:06.311865 (+ 5079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:06.311874 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:06.311889 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:06.312502 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:06.313059 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:06.313064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:06.313067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:06.313320 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:06.314469 (+ 1149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:06.314477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:06.315070 (+ 593us) client_negotiation.cc:769] Sending connection context
0809 06:22:06.317347 (+ 2277us) client_negotiation.cc:240] Negotiation successful
0809 06:22:06.317903 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14321,"thread_start_us":20716,"threads_started":1}
I20260809 06:22:06.319598 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:06.256245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:44213 (local address 127.3.137.126:46839)
0809 06:22:06.268846 (+ 12601us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:06.268873 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:06.278759 (+ 9886us) server_negotiation.cc:407] Connection header received
0809 06:22:06.282286 (+ 3527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:06.282306 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:06.282819 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:06.283284 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:06.285238 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.287209 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.289662 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.290878 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.291138 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:06.307241 (+ 16103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:06.307254 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:06.307276 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:06.307344 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:06.311652 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:06.312689 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:06.312698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:06.312705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:06.312909 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:06.313475 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:06.313482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:06.313486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:06.314196 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:06.314379 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:06.317611 (+ 3232us) server_negotiation.cc:299] Negotiation successful
0809 06:22:06.318414 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12227,"thread_start_us":11960,"threads_started":1}
I20260809 06:22:06.323433 17668 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:06.323783 17668 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:06.324676 17668 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
W20260809 06:22:06.328096 17668 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:46839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:22:06.328337 17668 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:06.328655 17668 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:06.330046 17668 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:06.330380 17668 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
W20260809 06:22:06.331507 17668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:22:06.685570 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:06.686082 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:06.686255 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:06.694687 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:06.694864 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:06.695087 17672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:06.706032 17672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:35909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=37561
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:06.707461 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:06.708781 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:06.726377 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:06.738198 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:06.743975 17672 server_base.cc:1034] running on GCE node
W20260809 06:22:06.744195 17690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:06.745115 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:06.754745 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:06.756080 17672 hybrid_clock.cc:648] HybridClock initialized: now 1786256526756000 us; error 71 us; skew 500 ppm
I20260809 06:22:06.756515 17672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:06.764227 17672 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:06.784845 17672 webserver.cc:460] Webserver started at http://127.3.137.66:37561/ using document root <none> and password file <none>
I20260809 06:22:06.785856 17672 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:06.786039 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:06.821084 17672 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.003s
I20260809 06:22:06.825022 17700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.827091 17672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 06:22:06.827349 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "7a12f397821b420eb649c92fbe2dc236"
format_stamp: "Formatted at 2026-08-09 06:22:03 on dist-test-slave-2w3w"
server_key: "0210d0726835c01a5d757e70c73e875b"
server_key_iv: "a532c572777e5dd17255358b7962d824"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:06.828778 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:06.853230 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:06.858804 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:06.859210 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:06.866628 17672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:06.877877 17672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:06.878005 17672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.878123 17672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:06.878209 17672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:06.945353 17672 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:35909
I20260809 06:22:06.945417 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:35909 every 8 connection(s)
I20260809 06:22:06.946712 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:06.948501 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17672
I20260809 06:22:06.949332 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.137.84:41015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:45183
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:46839
--webserver_port=36531
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:06 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:06.978243 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:06.953166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:39337 (local address 127.3.137.126:46839)
0809 06:22:06.953709 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:06.953717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:06.954889 (+ 1172us) server_negotiation.cc:407] Connection header received
0809 06:22:06.958026 (+ 3137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:06.958034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:06.958132 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:06.958217 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:06.960601 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.961824 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.964451 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.965146 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.965278 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:06.970060 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:06.970068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:06.970072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:06.970116 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:06.973932 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:06.975014 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:06.975020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:06.975023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:06.975193 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:06.975810 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:06.975815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:06.975818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:06.976217 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:06.976380 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:06.977544 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0809 06:22:06.977790 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260809 06:22:06.980846 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:06.953782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.66:39337)
0809 06:22:06.954673 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:06.954764 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:06.957666 (+ 2902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:06.958786 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:06.958800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:06.959506 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:06.960350 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:06.960378 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.962037 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.962044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:06.964235 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:06.964267 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:06.965694 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:06.965703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:06.966005 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:06.966867 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:06.966902 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:06.969751 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:06.974146 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:06.974155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:06.974174 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:06.974819 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:06.975382 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:06.975386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:06.975390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:06.975654 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:06.976481 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:06.976491 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:06.976986 (+ 495us) client_negotiation.cc:769] Sending connection context
0809 06:22:06.978628 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0809 06:22:06.979364 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":325,"threads_started":1}
I20260809 06:22:06.982640 17821 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:06.983003 17821 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:06.983938 17821 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:06.986001 17451 ts_manager.cc:194] Registered new tserver with Master: 7a12f397821b420eb649c92fbe2dc236 (127.3.137.66:35909)
I20260809 06:22:06.988184 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:39337
Aug 09 06:22:07 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:07.331674 17668 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:07.332149 17668 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:07.336596 17451 ts_manager.cc:194] Registered new tserver with Master: b8812cf5d6b6488bb8089b2280c4ecaf (127.3.137.65:43187)
I20260809 06:22:07.337828 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:44213
W20260809 06:22:07.412559 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:07.412964 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:07.413129 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:07.424659 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:07.424796 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:07.424985 17825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:07.436723 17825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.137.84:41015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:45183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=36531
--tserver_master_addrs=127.3.137.126:46839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:07.438052 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:07.439641 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:07.453214 17831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:07.453543 17830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:07.454336 17825 server_base.cc:1034] running on GCE node
W20260809 06:22:07.454509 17833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:07.455248 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:07.456239 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:07.457559 17825 hybrid_clock.cc:648] HybridClock initialized: now 1786256527457502 us; error 45 us; skew 500 ppm
I20260809 06:22:07.457849 17825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:07 dist-test-slave-2w3w krb5kdc[16367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:07.461395 17825 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:07.463411 17825 webserver.cc:460] Webserver started at http://127.3.137.67:36531/ using document root <none> and password file <none>
I20260809 06:22:07.464305 17825 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:07.464473 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:22:07 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:07.491925 17825 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:22:07.495684 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:07.497536 17825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:22:07.497706 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "9820e870d0984855ba6604994be21568"
format_stamp: "Formatted at 2026-08-09 06:22:04 on dist-test-slave-2w3w"
server_key: "bcc4a3017299ce19528577d3a57644be"
server_key_iv: "2ff1e3642ffef2251bdc1b281423e947"
server_key_version: "kuduclusterkey@0"
I20260809 06:22:07.499122 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:07.533521 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:07.538560 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:07.538884 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:07.540170 17825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:07.541934 17825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:07.542047 17825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:07.542160 17825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:07.542241 17825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:22:07 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for HTTP/127.3.137.84@KRBTEST.COM
I20260809 06:22:07.640528 17825 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:45183
I20260809 06:22:07.640609 17960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:45183 every 8 connection(s)
I20260809 06:22:07.642496 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:07.645856 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 17825
Aug 09 06:22:07 dist-test-slave-2w3w krb5kdc[16367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:07.715931 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.647824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.3.137.126:46839)
0809 06:22:07.678353 (+ 30529us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.678363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.679603 (+ 1240us) server_negotiation.cc:407] Connection header received
0809 06:22:07.679697 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.679706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.679838 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.679965 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:07.688530 (+ 8565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.689975 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.700664 (+ 10689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.701338 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.701444 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.705487 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:07.705494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:07.705499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:07.705542 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:07.709269 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.711323 (+ 2054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.711329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.711332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.711485 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.712020 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.712026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.712029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.712445 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:07.713640 (+ 1195us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.715295 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.715542 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30379,"thread_start_us":30249,"threads_started":1}
I20260809 06:22:07.719841 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.646477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:50055 (local address 127.3.137.126:46839)
0809 06:22:07.679157 (+ 32680us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.679166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.685234 (+ 6068us) server_negotiation.cc:407] Connection header received
0809 06:22:07.685314 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.685323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.685436 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.685557 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:07.702179 (+ 16622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.703495 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.706141 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.706985 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.707105 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.712010 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:07.712017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:07.712022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:07.712102 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:07.715718 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.716791 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.716799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.716802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.716938 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.717792 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.717797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.717800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.718134 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:07.718256 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.719292 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.719493 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31314,"spinlock_wait_cycles":2304,"thread_start_us":31173,"threads_started":1}
I20260809 06:22:07.722330 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.645874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:46839 (local address 127.3.137.67:50055)
0809 06:22:07.680933 (+ 35059us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:07.681015 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:07.684651 (+ 3636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:07.700240 (+ 15589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:07.700253 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:07.701018 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:07.701940 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.701971 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.703751 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.703760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.705961 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.705978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.707498 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.707508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.707835 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.708585 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:07.708624 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:07.711712 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:07.715967 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:07.715976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:07.715996 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:07.716643 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:07.717107 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:07.717111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:07.717114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:07.717651 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:07.718358 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:07.718365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:07.718827 (+ 462us) client_negotiation.cc:769] Sending connection context
0809 06:22:07.720111 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0809 06:22:07.720892 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34610,"thread_start_us":38979,"threads_started":1}
I20260809 06:22:07.723903 17961 heartbeater.cc:347] Connected to a master server at 127.3.137.126:46839
I20260809 06:22:07.724241 17961 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:07.725124 17961 heartbeater.cc:511] Master 127.3.137.126:46839 requested a full tablet report, sending...
I20260809 06:22:07.726792 17451 ts_manager.cc:194] Registered new tserver with Master: 9820e870d0984855ba6604994be21568 (127.3.137.67:45183)
I20260809 06:22:07.727941 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:50055
I20260809 06:22:07.738988 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:22:07.787724 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.767115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.3.137.126:46839)
0809 06:22:07.767291 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.767297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.770223 (+ 2926us) server_negotiation.cc:407] Connection header received
0809 06:22:07.770430 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.770436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.770521 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.770631 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:07.772914 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.774031 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.776133 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.776820 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.776924 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.780438 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:07.780443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:07.780446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:07.780478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:07.783680 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.784638 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.784643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.784646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.784813 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:07.785290 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:07.785295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:07.785298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:07.785653 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:07.785916 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.787138 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.787358 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:22:07.794116 17451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:22:07.805552 17451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:22:07.879673 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.856839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:45918)
0809 06:22:07.857552 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:07.857582 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:07.857738 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:07.867659 (+ 9921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:07.867665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:07.867694 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:07.868173 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.868188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.872501 (+ 4313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.872509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.874663 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.874678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.878466 (+ 3788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.878474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.878722 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.878793 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 06:22:07.879012 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 06:22:07.879233 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":1,"thread_start_us":386,"threads_started":1}
I20260809 06:22:07.880820 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.852004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.3.137.67:45183)
0809 06:22:07.857193 (+ 5189us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.857201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.857237 (+ 36us) server_negotiation.cc:407] Connection header received
0809 06:22:07.857337 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.857344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.857592 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.857808 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:07.863077 (+ 5269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.865354 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.869177 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.870946 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.871138 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.871450 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.880272 (+ 8822us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.880467 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5018,"thread_start_us":5045,"threads_started":1}
I20260809 06:22:07.881738 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.851604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45183 (local address 127.0.0.1:37156)
0809 06:22:07.855523 (+ 3919us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:07.855560 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:07.855752 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:07.858169 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:07.858176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:07.858212 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:07.858750 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.858773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.865585 (+ 6812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.865594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.868921 (+ 3327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.868943 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.878638 (+ 9695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.878648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.880051 (+ 1403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.880083 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:22:07.881205 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0809 06:22:07.881352 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3665,"mutex_wait_us":1198,"thread_start_us":3532,"threads_started":1}
I20260809 06:22:07.882774 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.856946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.3.137.65:43187)
0809 06:22:07.866141 (+ 9195us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.866151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.866183 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:22:07.866281 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.866289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.866540 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.868450 (+ 1910us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:07.868712 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.870291 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.875122 (+ 4831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.876585 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.882020 (+ 5435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.882255 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.882319 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.882472 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9023,"thread_start_us":8339,"threads_started":1}
I20260809 06:22:07.885954 17891 tablet_service.cc:1467] Processing CreateTablet for tablet 341ddd95364747aa83b77deab08e34b1 (DEFAULT_TABLE table=test-table [id=16768f9f04804862b7e65779f1d77296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:07.889230 17597 tablet_service.cc:1467] Processing CreateTablet for tablet 341ddd95364747aa83b77deab08e34b1 (DEFAULT_TABLE table=test-table [id=16768f9f04804862b7e65779f1d77296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:07.891810 17597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341ddd95364747aa83b77deab08e34b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:07.892686 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341ddd95364747aa83b77deab08e34b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:07.893512 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.858127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:35909 (local address 127.0.0.1:52758)
0809 06:22:07.879144 (+ 21017us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:07.879159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:07.879251 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:07.886394 (+ 7143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:07.886399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:07.886425 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:07.886730 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.886743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.888866 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.888872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.890908 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:07.890924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.892768 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.892775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:07.892957 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.892989 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:22:07.893055 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 06:22:07.893178 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20901,"thread_start_us":20735,"threads_started":1}
I20260809 06:22:07.894595 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:07.858385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.3.137.66:35909)
0809 06:22:07.885618 (+ 27233us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:07.885626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:07.885654 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:22:07.885759 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:07.885767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:07.886000 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:07.886937 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:07.887179 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.888691 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.891141 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:07.892511 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:07.893837 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:07.894045 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:07.894117 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 06:22:07.894289 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27069,"thread_start_us":27761,"threads_started":1}
I20260809 06:22:07.898523 17750 tablet_service.cc:1467] Processing CreateTablet for tablet 341ddd95364747aa83b77deab08e34b1 (DEFAULT_TABLE table=test-table [id=16768f9f04804862b7e65779f1d77296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:07.899966 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341ddd95364747aa83b77deab08e34b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:07.920127 17984 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Bootstrap starting.
I20260809 06:22:07.922204 17983 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Bootstrap starting.
I20260809 06:22:07.926218 17984 tablet_bootstrap.cc:654] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:07.931358 17983 tablet_bootstrap.cc:654] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:07.931452 17984 log.cc:824] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:07.932137 17982 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Bootstrap starting.
I20260809 06:22:07.936244 17982 tablet_bootstrap.cc:654] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:07.939903 17984 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: No bootstrap required, opened a new log
I20260809 06:22:07.940208 17984 ts_tablet_manager.cc:1397] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20260809 06:22:07.941059 17983 log.cc:824] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:07.937628 17982 log.cc:824] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:07.952487 17982 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: No bootstrap required, opened a new log
I20260809 06:22:07.952852 17982 ts_tablet_manager.cc:1397] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20260809 06:22:07.954779 17983 tablet_bootstrap.cc:492] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: No bootstrap required, opened a new log
I20260809 06:22:07.955161 17983 ts_tablet_manager.cc:1397] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.005s
I20260809 06:22:07.961998 17984 raft_consensus.cc:348] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.966203 17982 raft_consensus.cc:348] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.966675 17982 raft_consensus.cc:374] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:07.966799 17982 raft_consensus.cc:729] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a12f397821b420eb649c92fbe2dc236, State: Initialized, Role: FOLLOWER
I20260809 06:22:07.967032 17984 raft_consensus.cc:374] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:07.967803 17984 raft_consensus.cc:729] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8812cf5d6b6488bb8089b2280c4ecaf, State: Initialized, Role: FOLLOWER
I20260809 06:22:07.968742 17984 consensus_queue.cc:260] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.968802 17982 consensus_queue.cc:260] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.968735 17983 raft_consensus.cc:348] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.969206 17983 raft_consensus.cc:374] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:07.969403 17983 raft_consensus.cc:729] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9820e870d0984855ba6604994be21568, State: Initialized, Role: FOLLOWER
I20260809 06:22:07.970279 17983 consensus_queue.cc:260] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:07.974134 17668 heartbeater.cc:503] Master 127.3.137.126:46839 was elected leader, sending a full tablet report...
I20260809 06:22:07.975232 17984 ts_tablet_manager.cc:1428] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Time spent starting tablet: real 0.035s user 0.009s sys 0.009s
I20260809 06:22:07.976106 17821 heartbeater.cc:503] Master 127.3.137.126:46839 was elected leader, sending a full tablet report...
I20260809 06:22:07.976828 17983 ts_tablet_manager.cc:1428] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Time spent starting tablet: real 0.021s user 0.012s sys 0.004s
I20260809 06:22:07.977169 17961 heartbeater.cc:503] Master 127.3.137.126:46839 was elected leader, sending a full tablet report...
I20260809 06:22:07.978374 17982 ts_tablet_manager.cc:1428] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Time spent starting tablet: real 0.023s user 0.009s sys 0.009s
W20260809 06:22:08.020897 17669 tablet.cc:2369] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:08.042132 17993 raft_consensus.cc:482] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:22:08.042497 17993 raft_consensus.cc:504] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:08.044637 17993 leader_election.cc:288] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9820e870d0984855ba6604994be21568 (127.3.137.67:45183), b8812cf5d6b6488bb8089b2280c4ecaf (127.3.137.65:43187)
I20260809 06:22:08.052979 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.045534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:45269 (local address 127.3.137.67:45183)
0809 06:22:08.045694 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.045699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.046749 (+ 1050us) server_negotiation.cc:407] Connection header received
0809 06:22:08.046944 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.046949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.047060 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.047114 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:08.047898 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.048921 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.051064 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.051997 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.052094 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.052145 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.052583 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.052711 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:22:08.053117 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.045038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:45183 (local address 127.3.137.66:45269)
0809 06:22:08.046626 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.046650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:08.046777 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:08.047274 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:08.047277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:08.047299 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:08.047702 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:08.047714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.049065 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.049070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:08.050850 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:08.050863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.052296 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.052300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:08.052416 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.052437 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 06:22:08.052551 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 06:22:08.052675 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"mutex_wait_us":6,"thread_start_us":1291,"threads_started":1}
I20260809 06:22:08.053579 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.045166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:60009 (local address 127.3.137.65:43187)
0809 06:22:08.045366 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.045372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.046294 (+ 922us) server_negotiation.cc:407] Connection header received
0809 06:22:08.046654 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.046661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.046765 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.046856 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:08.047536 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.048698 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.051211 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.052356 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.052485 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.052554 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.053208 (+ 654us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.053357 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:22:08.054342 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.045001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.3.137.66:60009)
0809 06:22:08.046179 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.046201 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:08.046438 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:08.046895 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:08.046899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:08.046925 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:08.047323 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:08.047333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.048835 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.048840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:08.051037 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:08.051051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.052758 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.052764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:08.052928 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.052962 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:22:08.053892 (+ 930us) client_negotiation.cc:240] Negotiation successful
0809 06:22:08.054009 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":856,"threads_started":1}
I20260809 06:22:08.054538 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341ddd95364747aa83b77deab08e34b1" candidate_uuid: "7a12f397821b420eb649c92fbe2dc236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9820e870d0984855ba6604994be21568" is_pre_election: true
I20260809 06:22:08.055233 17911 raft_consensus.cc:2454] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12f397821b420eb649c92fbe2dc236 in term 0.
I20260809 06:22:08.055438 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341ddd95364747aa83b77deab08e34b1" candidate_uuid: "7a12f397821b420eb649c92fbe2dc236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" is_pre_election: true
I20260809 06:22:08.055922 17702 leader_election.cc:302] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a12f397821b420eb649c92fbe2dc236, 9820e870d0984855ba6604994be21568; no voters:
I20260809 06:22:08.056037 17618 raft_consensus.cc:2454] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a12f397821b420eb649c92fbe2dc236 in term 0.
I20260809 06:22:08.056366 17993 raft_consensus.cc:2781] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:22:08.056526 17993 raft_consensus.cc:482] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:22:08.056624 17993 raft_consensus.cc:3037] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:08.059840 17993 raft_consensus.cc:504] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:08.060498 17993 leader_election.cc:288] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [CANDIDATE]: Term 1 election: Requested vote from peers 9820e870d0984855ba6604994be21568 (127.3.137.67:45183), b8812cf5d6b6488bb8089b2280c4ecaf (127.3.137.65:43187)
I20260809 06:22:08.060902 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341ddd95364747aa83b77deab08e34b1" candidate_uuid: "7a12f397821b420eb649c92fbe2dc236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf"
I20260809 06:22:08.060945 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341ddd95364747aa83b77deab08e34b1" candidate_uuid: "7a12f397821b420eb649c92fbe2dc236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9820e870d0984855ba6604994be21568"
I20260809 06:22:08.061105 17618 raft_consensus.cc:3037] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:08.061144 17911 raft_consensus.cc:3037] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:08.063398 17618 raft_consensus.cc:2454] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12f397821b420eb649c92fbe2dc236 in term 1.
I20260809 06:22:08.063833 17701 leader_election.cc:302] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a12f397821b420eb649c92fbe2dc236, b8812cf5d6b6488bb8089b2280c4ecaf; no voters:
I20260809 06:22:08.064131 17993 raft_consensus.cc:2781] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:08.064347 17911 raft_consensus.cc:2454] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a12f397821b420eb649c92fbe2dc236 in term 1.
I20260809 06:22:08.064685 17993 raft_consensus.cc:686] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 1 LEADER]: Becoming Leader. State: Replica: 7a12f397821b420eb649c92fbe2dc236, State: Running, Role: LEADER
I20260809 06:22:08.065261 17993 consensus_queue.cc:237] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } }
I20260809 06:22:08.070283 17451 catalog_manager.cc:5495] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a12f397821b420eb649c92fbe2dc236 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "7a12f397821b420eb649c92fbe2dc236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a12f397821b420eb649c92fbe2dc236" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 35909 } health_report { overall_health: HEALTHY } } }
I20260809 06:22:08.110215 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.104379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.3.137.66:35909)
0809 06:22:08.104572 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.104578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.104651 (+ 73us) server_negotiation.cc:407] Connection header received
0809 06:22:08.104938 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.104943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.105069 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.105229 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:22:08.105639 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.106401 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.107975 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.108376 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.108515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.109063 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:22:08.109558 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:22:08.109717 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.109803 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.109969 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:22:08.118409 17911 raft_consensus.cc:1261] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 1 FOLLOWER]: Refusing update from remote peer 7a12f397821b420eb649c92fbe2dc236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:08.118422 17618 raft_consensus.cc:1261] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 1 FOLLOWER]: Refusing update from remote peer 7a12f397821b420eb649c92fbe2dc236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:08.119465 18000 consensus_queue.cc:1035] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8812cf5d6b6488bb8089b2280c4ecaf" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:08.119889 17993 consensus_queue.cc:1035] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9820e870d0984855ba6604994be21568" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:08.138648 18003 mvcc.cc:205] Tried to move back new op lower bound from 7316506739162566656 to 7316506738959818752. Current Snapshot: MvccSnapshot[applied={T|T < 7316506739162566656}]
I20260809 06:22:08.140811 18006 mvcc.cc:205] Tried to move back new op lower bound from 7316506739162566656 to 7316506738959818752. Current Snapshot: MvccSnapshot[applied={T|T < 7316506739162566656}]
I20260809 06:22:08.147961 18005 mvcc.cc:205] Tried to move back new op lower bound from 7316506739162566656 to 7316506738959818752. Current Snapshot: MvccSnapshot[applied={T|T < 7316506739162566656}]
I20260809 06:22:08.156411 17451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:08.156675 17451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:08.161586 17451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 374a45eaef93464d8b42fabe9fda4ba9: Sending DeleteTablet for 3 replicas of tablet 341ddd95364747aa83b77deab08e34b1
I20260809 06:22:08.163501 17750 tablet_service.cc:1514] Processing DeleteTablet for tablet 341ddd95364747aa83b77deab08e34b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:08 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:52758
I20260809 06:22:08.163645 17597 tablet_service.cc:1514] Processing DeleteTablet for tablet 341ddd95364747aa83b77deab08e34b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:08 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:45918
I20260809 06:22:08.164320 17891 tablet_service.cc:1514] Processing DeleteTablet for tablet 341ddd95364747aa83b77deab08e34b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:08 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:37156
I20260809 06:22:08.172811 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17490
I20260809 06:22:08.190141 18016 tablet_replica.cc:330] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: stopping tablet replica
I20260809 06:22:08.191331 18016 raft_consensus.cc:2229] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:08.192602 18016 raft_consensus.cc:2258] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:08.193594 18014 tablet_replica.cc:330] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: stopping tablet replica
I20260809 06:22:08.195158 18014 raft_consensus.cc:2229] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:22:08.196777 18016 ts_tablet_manager.cc:1905] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:08.197863 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.174325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.3.137.65:43187)
0809 06:22:08.174485 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.174491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.174542 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:22:08.174767 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.174775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.174867 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.175008 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:08.178404 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.179997 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.182179 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.183162 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.183289 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.187426 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:08.187448 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:08.187473 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:08.187526 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:08.192377 (+ 4851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.193582 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.193591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.193594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.193828 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.195052 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.195057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.195060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.195537 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:08.195828 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.197132 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.197441 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:22:08.199102 17663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:08.199187 18014 raft_consensus.cc:2258] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:08.198770 18015 tablet_replica.cc:330] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf: stopping tablet replica
I20260809 06:22:08.202028 18014 ts_tablet_manager.cc:1905] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:08.206131 18016 ts_tablet_manager.cc:1918] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:08.206357 18016 log.cc:1196] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/341ddd95364747aa83b77deab08e34b1
I20260809 06:22:08.207010 18016 ts_tablet_manager.cc:1939] T 341ddd95364747aa83b77deab08e34b1 P 9820e870d0984855ba6604994be21568: Deleting consensus metadata
I20260809 06:22:08.208881 17436 catalog_manager.cc:4841] TS 9820e870d0984855ba6604994be21568 (127.3.137.67:45183): tablet 341ddd95364747aa83b77deab08e34b1 (table test-table [id=16768f9f04804862b7e65779f1d77296]) successfully deleted
I20260809 06:22:08.211410 18014 ts_tablet_manager.cc:1918] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:08.211700 18014 log.cc:1196] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/341ddd95364747aa83b77deab08e34b1
I20260809 06:22:08.212440 18014 ts_tablet_manager.cc:1939] T 341ddd95364747aa83b77deab08e34b1 P 7a12f397821b420eb649c92fbe2dc236: Deleting consensus metadata
I20260809 06:22:08.214579 17435 catalog_manager.cc:4841] TS 7a12f397821b420eb649c92fbe2dc236 (127.3.137.66:35909): tablet 341ddd95364747aa83b77deab08e34b1 (table test-table [id=16768f9f04804862b7e65779f1d77296]) successfully deleted
I20260809 06:22:08.200062 18015 raft_consensus.cc:2229] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:08.304805 18015 raft_consensus.cc:2258] T 341ddd95364747aa83b77deab08e34b1 P b8812cf5d6b6488bb8089b2280c4ecaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:08.313146 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17490
W20260809 06:22:08.334262 17435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:43187 (error 108)
I20260809 06:22:08.335270 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17672
I20260809 06:22:08.350829 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.350278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:45928)
0809 06:22:08.350483 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.350654 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:22:08.351274 17435 catalog_manager.cc:4584] TS b8812cf5d6b6488bb8089b2280c4ecaf (127.3.137.65:43187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 341ddd95364747aa83b77deab08e34b1: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
I20260809 06:22:08.356637 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.337299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.3.137.66:35909)
0809 06:22:08.337537 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.337545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.337568 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:22:08.337653 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.337661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.337761 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.337877 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:08.340513 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.342156 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.344656 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.345568 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.345720 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.350169 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:08.350183 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:08.350205 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:08.350260 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:08.353631 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.354382 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.354388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.354391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.354558 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.354992 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.355000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.355006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.355380 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:08.355720 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.356025 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.356248 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 06:22:08.357405 17816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:08.405251 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.404806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58882)
0809 06:22:08.405006 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.405113 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:22:08.406322 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.406007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58894)
0809 06:22:08.406132 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.406203 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 06:22:08.467154 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.466409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58906)
0809 06:22:08.466774 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.466886 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260809 06:22:08.468801 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.468393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58912)
0809 06:22:08.468554 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.468653 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 06:22:08.468969 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17672
I20260809 06:22:08.490484 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17825
I20260809 06:22:08.504057 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.491228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.3.137.67:45183)
0809 06:22:08.491414 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:08.491420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:08.491521 (+ 101us) server_negotiation.cc:407] Connection header received
0809 06:22:08.491729 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:08.491734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:08.491795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:08.491907 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:08.493249 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.493959 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.495326 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:08.495700 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:08.495830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:08.498341 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:08.498353 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:08.498368 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:08.498413 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:08.501097 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.501857 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.501871 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.501873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.502029 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:08.502423 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:08.502428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:08.502431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:08.502776 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:08.503124 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:08.503427 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 06:22:08.503648 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:22:08.504712 17956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:08.549535 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.549054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58922)
0809 06:22:08.549260 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.549357 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 06:22:08.550604 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.550226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58936)
0809 06:22:08.550413 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.550486 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260809 06:22:08.596345 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17825
I20260809 06:22:08.615655 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 17417
I20260809 06:22:08.616932 17483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:08.731379 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:08.730695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43187 (local address 127.0.0.1:58940)
0809 06:22:08.730898 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:08.731028 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 06:22:08.731755 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 17417
I20260809 06:22:08.760268 3621 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 06:22:09.105182 3621 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 06:22:09.105398 3621 mini_ranger.cc:66] Stopping Ranger...
I20260809 06:22:09.449911 3621 mini_ranger.cc:68] Stopped Ranger
2026-08-09 06:22:09.450 UTC [16398] LOG: received smart shutdown request
2026-08-09 06:22:09.453 UTC [16398] LOG: background worker "logical replication launcher" (PID 16407) exited with exit code 1
2026-08-09 06:22:09.454 UTC [16402] LOG: shutting down
2026-08-09 06:22:09.619 UTC [16398] LOG: database system is shut down
2026-08-09T06:22:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52267 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:22:09 dist-test-slave-2w3w krb5kdc[18031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:22:09 dist-test-slave-2w3w krb5kdc[18031](info): set up 2 sockets
Aug 09 06:22:09 dist-test-slave-2w3w krb5kdc[18031](info): commencing operation
krb5kdc: starting...
W20260809 06:22:10.900977 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:22:11 dist-test-slave-2w3w krb5kdc[18031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:22:11.178576 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43659
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:11.511449 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:11.511732 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:11.511822 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:11.518919 18044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:11.519104 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:11.519170 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:11.519224 18044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:11.519270 18044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:11.529131 18044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:11.530371 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:11.531728 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:11.543592 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:11.543561 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:11.543637 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:11.544430 18044 server_base.cc:1034] running on GCE node
I20260809 06:22:11.545168 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:11.546502 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:11.547701 18044 hybrid_clock.cc:648] HybridClock initialized: now 1786256531547663 us; error 34 us; skew 500 ppm
I20260809 06:22:11.548041 18044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:11 dist-test-slave-2w3w krb5kdc[18031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:11.551182 18044 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:11.552659 18044 webserver.cc:460] Webserver started at http://127.3.137.126:34437/ using document root <none> and password file <none>
I20260809 06:22:11.553275 18044 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:11.553372 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:11.553623 18044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:11.556555 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "e0b63e98307e487ca3c74d812f82926e"
format_stamp: "Formatted at 2026-08-09 06:22:11 on dist-test-slave-2w3w"
server_key: "adaf9ab73e0dae4a31b4672a92ded4d4"
server_key_iv: "fb259a81e48d674bd9f93aa8624b5d8f"
server_key_version: "encryptionkey@0"
I20260809 06:22:11.557314 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "e0b63e98307e487ca3c74d812f82926e"
format_stamp: "Formatted at 2026-08-09 06:22:11 on dist-test-slave-2w3w"
server_key: "adaf9ab73e0dae4a31b4672a92ded4d4"
server_key_iv: "fb259a81e48d674bd9f93aa8624b5d8f"
server_key_version: "encryptionkey@0"
I20260809 06:22:11.562269 18044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:22:11.565538 18060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:11.566869 18044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:22:11.567088 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "e0b63e98307e487ca3c74d812f82926e"
format_stamp: "Formatted at 2026-08-09 06:22:11 on dist-test-slave-2w3w"
server_key: "adaf9ab73e0dae4a31b4672a92ded4d4"
server_key_iv: "fb259a81e48d674bd9f93aa8624b5d8f"
server_key_version: "encryptionkey@0"
I20260809 06:22:11.567296 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:11.592495 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:11.596560 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:11.596870 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:11.616101 18044 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:43659
I20260809 06:22:11.616145 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:43659 every 8 connection(s)
I20260809 06:22:11.617478 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:11.621138 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:11.623247 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18044
I20260809 06:22:11.623487 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:22:11.624423 3621 external_mini_cluster.cc:1428] Setting key 8785b09d142784601b9e4d00b8f4fefe
I20260809 06:22:11.630908 18119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:22:11 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.125@KRBTEST.COM" created.
I20260809 06:22:11.653903 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.634791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.3.137.126:43659)
0809 06:22:11.636556 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:11.636564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:11.636615 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:22:11.638134 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:11.638139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:11.638354 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:11.638567 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:11.639858 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:11.641056 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:11.642422 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:11.643024 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:11.643228 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:11.646793 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:11.646801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:11.646813 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:11.646841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:11.649639 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:11.650312 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:11.650321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:11.650325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:11.650477 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:11.650904 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:11.650912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:11.650915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:11.651428 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:11.651651 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:11.652164 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 06:22:11.652489 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1543,"spinlock_wait_cycles":53632,"thread_start_us":1308,"threads_started":1}
I20260809 06:22:11.653947 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.634780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43659 (local address 127.0.0.1:56292)
0809 06:22:11.635710 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.635757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:11.637755 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:11.638582 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:11.638588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:11.639324 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:11.639655 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:11.639673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:11.641214 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:11.641219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:11.642270 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:11.642283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:11.643254 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:11.643261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:11.643443 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:11.644140 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:11.644162 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:11.646548 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:11.649768 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:11.649782 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:11.649794 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:11.650180 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:11.650631 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:11.650636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:11.650639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:11.650796 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:11.651578 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:11.651585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:11.651799 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 06:22:11.652047 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 06:22:11.652601 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":152,"thread_start_us":461,"threads_started":1}
I20260809 06:22:11.656790 18119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:22:11.657797 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.657420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44952)
0809 06:22:11.657591 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.657695 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 06:22:11.658113 18064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
I20260809 06:22:11.659688 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.659355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44954)
0809 06:22:11.659507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.659573 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 06:22:11.659992 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:41065: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
I20260809 06:22:11.685819 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:41065
--webserver_interface=127.3.137.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:11.709319 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } attempt: 1
I20260809 06:22:11.710597 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.710117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44970)
0809 06:22:11.710328 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.710446 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:22:11.711719 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.711387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44972)
0809 06:22:11.711529 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.711598 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:22:11.711952 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:41065: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
I20260809 06:22:11.780193 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } attempt: 2
I20260809 06:22:11.781379 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.780914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44982)
0809 06:22:11.781151 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.781242 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 06:22:11.782601 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.782243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44990)
0809 06:22:11.782382 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.782472 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 06:22:11.782841 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:41065: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
I20260809 06:22:11.874090 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } attempt: 3
I20260809 06:22:11.875404 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.874911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44992)
0809 06:22:11.875163 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.875271 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 06:22:11.876518 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:11.876159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:44998)
0809 06:22:11.876296 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:11.876364 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 06:22:11.876763 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:41065: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
I20260809 06:22:12.020015 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } attempt: 4
I20260809 06:22:12.021297 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.020758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:45000)
0809 06:22:12.021072 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.021161 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260809 06:22:12.022265 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.021931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:45006)
0809 06:22:12.022063 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.022150 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 06:22:12.022506 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:41065: Network error: Client connection negotiation failed: client connection to 127.3.137.125:41065: connect: Connection refused (error 111)
W20260809 06:22:12.022444 18126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:12.022812 18126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:12.022992 18126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:12.030213 18126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:12.030337 18126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:12.030447 18126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:12.030550 18126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:12.030629 18126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:12.040861 18126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:41065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.137.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:12.042126 18126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:12.043497 18126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:12.055114 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:12.055202 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:12.055471 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:12.055996 18126 server_base.cc:1034] running on GCE node
I20260809 06:22:12.056743 18126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:12.058023 18126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:12.059222 18126 hybrid_clock.cc:648] HybridClock initialized: now 1786256532059184 us; error 34 us; skew 500 ppm
I20260809 06:22:12.059573 18126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:12.065596 18126 init.cc:377] Logged in from keytab as kudu/127.3.137.125@KRBTEST.COM (short username kudu)
I20260809 06:22:12.067157 18126 webserver.cc:460] Webserver started at http://127.3.137.125:38223/ using document root <none> and password file <none>
I20260809 06:22:12.067862 18126 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:12.068017 18126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:12.068331 18126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:12.071691 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data/instance:
uuid: "6e1b72bb4e7046648aa394c252fc0291"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "0cfa59918ab981178315332b1c09895c"
server_key_iv: "54ae347fdde2ae8c8d5e04ca7b9607f1"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.072583 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal/instance:
uuid: "6e1b72bb4e7046648aa394c252fc0291"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "0cfa59918ab981178315332b1c09895c"
server_key_iv: "54ae347fdde2ae8c8d5e04ca7b9607f1"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.077623 18126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:22:12.080912 18141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:12.082379 18126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:22:12.082576 18126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "6e1b72bb4e7046648aa394c252fc0291"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "0cfa59918ab981178315332b1c09895c"
server_key_iv: "54ae347fdde2ae8c8d5e04ca7b9607f1"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.082770 18126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:12.102478 18126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:12.106308 18126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:12.106595 18126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:12.125309 18126 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.125:41065
I20260809 06:22:12.125356 18195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.125:41065 every 8 connection(s)
I20260809 06:22:12.126730 18126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
I20260809 06:22:12.130473 18196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:12.132664 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18126
I20260809 06:22:12.132890 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-1/wal/instance
I20260809 06:22:12.133466 3621 external_mini_cluster.cc:1428] Setting key 26d073bba093ab3da93f19013623a376
I20260809 06:22:12.139711 18196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.124@KRBTEST.COM; defaulting to no policy
I20260809 06:22:12.161906 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.143060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.3.137.126:43659)
0809 06:22:12.143278 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.143285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.144465 (+ 1180us) server_negotiation.cc:407] Connection header received
0809 06:22:12.146438 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.146446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.146547 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.146662 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.148110 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.149277 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.150764 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.151574 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.151727 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.154989 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.154999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.155007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.155054 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.158866 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.159641 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.159650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.159656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.159805 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.160222 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.160228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.160231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.160670 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.160823 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.161507 (+ 684us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.161674 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:22:12.162822 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.143483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43659 (local address 127.0.0.1:56304)
0809 06:22:12.144274 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.144330 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.146168 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.146867 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.146874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.147413 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.147933 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.147950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.149435 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.149439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.150615 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.150627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.151770 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.151775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.151960 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.152647 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.152668 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.154732 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.159062 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.159068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.159080 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.159511 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.159925 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.159928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.159930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.160090 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.160796 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.160801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.161145 (+ 344us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.161375 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.161707 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.124@KRBTEST.COM" created.
I20260809 06:22:12.164693 18196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:22:12.180322 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.165825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.3.137.125:41065)
0809 06:22:12.166470 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.166476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.166500 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:22:12.166539 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.166545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.166705 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.166853 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.167607 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.168645 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.169765 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.170389 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.170539 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.173583 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.173594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.173605 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.173637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.177334 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.177957 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.177967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.177974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.178135 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.178510 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.178518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.178524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.179092 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.179253 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.179670 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.180009 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260809 06:22:12.180402 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.165563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:45022)
0809 06:22:12.165733 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.165752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.165854 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.166847 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.166851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.167323 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.167451 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.167461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.168781 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.168789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.169618 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.169628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.170544 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.170550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.170745 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.171368 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.171384 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.173374 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.177469 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.177472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.177475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.177823 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.178263 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.178267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.178269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.178397 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.179220 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.179228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.179388 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.179549 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.179881 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 06:22:12.182880 18196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:22:12.184114 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.183535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42886)
0809 06:22:12.183741 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.183911 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 06:22:12.184376 18142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.186129 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.185643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42900)
0809 06:22:12.185846 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.185952 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 06:22:12.186439 18196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
I20260809 06:22:12.197489 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:39449
--webserver_interface=127.3.137.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:12.235800 18196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 1
I20260809 06:22:12.237061 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.236520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42910)
0809 06:22:12.236741 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.236872 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 06:22:12.238090 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.237700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42916)
0809 06:22:12.237872 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.237952 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 06:22:12.238322 18196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.306572 18196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 2
I20260809 06:22:12.307782 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.307252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42930)
0809 06:22:12.307480 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.307592 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:22:12.308831 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.308441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42942)
0809 06:22:12.308610 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.308692 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 06:22:12.309060 18196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.321858 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } attempt: 5
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:22:12.335484 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.322611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.3.137.125:41065)
0809 06:22:12.322770 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.322777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.322821 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:22:12.323060 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.323072 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.323135 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.323237 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.324048 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.324755 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.325935 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.326401 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.326540 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.330104 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.330109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.330113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.330146 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.332841 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.333697 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.333701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.333704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.333814 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.334220 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.334223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.334225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.334480 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.334588 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.335105 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.335269 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:22:12.335681 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.322519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:45028)
0809 06:22:12.322735 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.322757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.322896 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.323282 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.323287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.323707 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.323891 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.323900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.324894 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.324902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.325787 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.325798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.326566 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.326572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.326756 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.327466 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.327484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.329917 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.332989 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.332995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.333000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.333576 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.333933 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.333938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.333941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.334113 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.334599 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.334604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.334794 (+ 190us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.334953 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.335316 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 06:22:12.336889 18119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:22:12.338136 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.337644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42944)
0809 06:22:12.337860 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.337951 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 06:22:12.340073 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.338736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42950)
0809 06:22:12.339742 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.339877 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":731,"threads_started":1}
W20260809 06:22:12.340305 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.391527 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 1
I20260809 06:22:12.392751 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.392285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42960)
0809 06:22:12.392524 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.392614 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 06:22:12.393838 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.393496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42974)
0809 06:22:12.393650 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.393717 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:22:12.394091 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.400290 18196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 3
I20260809 06:22:12.401364 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.400835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42988)
0809 06:22:12.401046 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.401153 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 06:22:12.402527 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.402048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:42992)
0809 06:22:12.402286 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.402391 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260809 06:22:12.402756 18196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.462337 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 2
I20260809 06:22:12.463568 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.463119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43008)
0809 06:22:12.463313 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.463428 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 06:22:12.464520 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.464178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43022)
0809 06:22:12.464321 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.464399 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:22:12.464706 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
W20260809 06:22:12.528573 18203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:12.528906 18203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:12.529066 18203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:12.536317 18203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:12.536459 18203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:12.536572 18203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:12.536662 18203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:12.536731 18203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:12.545998 18196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 4
I20260809 06:22:12.546365 18203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43659,127.3.137.125:41065,127.3.137.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.137.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:12.547386 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.546747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43038)
0809 06:22:12.547071 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.547179 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 06:22:12.548112 18203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:12.548425 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.548051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43042)
0809 06:22:12.548214 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.548298 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:22:12.548677 18196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.549528 18203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:12.560671 18209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:12.560917 18210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:12.561218 18212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:12.562103 18203 server_base.cc:1034] running on GCE node
I20260809 06:22:12.562826 18203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:12.564146 18203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:12.565371 18203 hybrid_clock.cc:648] HybridClock initialized: now 1786256532565330 us; error 48 us; skew 500 ppm
I20260809 06:22:12.565688 18203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:12.568674 18203 init.cc:377] Logged in from keytab as kudu/127.3.137.124@KRBTEST.COM (short username kudu)
I20260809 06:22:12.570045 18203 webserver.cc:460] Webserver started at http://127.3.137.124:46453/ using document root <none> and password file <none>
I20260809 06:22:12.570788 18203 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:12.570957 18203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:12.570916 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 3
I20260809 06:22:12.571331 18203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:12.571961 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.571513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43054)
0809 06:22:12.571703 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.571798 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 06:22:12.573019 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.572597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43070)
0809 06:22:12.572784 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.572861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260809 06:22:12.573217 18119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:39449: Network error: Client connection negotiation failed: client connection to 127.3.137.124:39449: connect: Connection refused (error 111)
I20260809 06:22:12.575197 18203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data/instance:
uuid: "bd335c70200249fd89d0181c75abbd6e"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "2dd4f71928282b9279563789c1c7ba3f"
server_key_iv: "5a891a7a46ace6da23e2d7f32eb2af93"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.576040 18203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal/instance:
uuid: "bd335c70200249fd89d0181c75abbd6e"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "2dd4f71928282b9279563789c1c7ba3f"
server_key_iv: "5a891a7a46ace6da23e2d7f32eb2af93"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.581079 18203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:22:12.584398 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:12.585629 18203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:22:12.585796 18203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "bd335c70200249fd89d0181c75abbd6e"
format_stamp: "Formatted at 2026-08-09 06:22:12 on dist-test-slave-2w3w"
server_key: "2dd4f71928282b9279563789c1c7ba3f"
server_key_iv: "5a891a7a46ace6da23e2d7f32eb2af93"
server_key_version: "encryptionkey@0"
I20260809 06:22:12.585995 18203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:12.635744 18203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:12.640738 18203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:12.641049 18203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:12.659467 18203 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.124:39449
I20260809 06:22:12.659513 18274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.124:39449 every 8 connection(s)
I20260809 06:22:12.660781 18203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
I20260809 06:22:12.664484 18275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:12.665099 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18203
I20260809 06:22:12.665306 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786256324325922-3621-0/minicluster-data/master-2/wal/instance
I20260809 06:22:12.665809 3621 external_mini_cluster.cc:1428] Setting key 07fedd33020201b8537c1da3ebed9015
I20260809 06:22:12.673120 18275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:12.688711 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.667819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.3.137.124:39449)
0809 06:22:12.668601 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.668611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.668654 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:22:12.670447 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.670469 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.670835 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.671326 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.672775 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.674134 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.675972 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.676906 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.677099 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.681156 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.681164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.681173 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.681206 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.684554 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.685253 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.685260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.685266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.685480 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.685944 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.685948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.685950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.686562 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.686818 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.687392 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.687707 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":370,"threads_started":1}
I20260809 06:22:12.691082 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.675533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43659 (local address 127.0.0.1:56316)
0809 06:22:12.676415 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.676462 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.676652 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.677052 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.677058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.677808 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.678127 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.678137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.679284 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.679289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.680303 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.680318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.681145 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.681154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.681310 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.681944 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.681967 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.684598 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.688250 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.688263 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.688270 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.688713 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.689172 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.689176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.689186 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.689326 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.689872 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.689879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.690098 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.690341 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.690641 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":554,"threads_started":1}
I20260809 06:22:12.691195 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.675518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.3.137.126:43659)
0809 06:22:12.676147 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.676154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.676565 (+ 411us) server_negotiation.cc:407] Connection header received
0809 06:22:12.676795 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.676799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.676864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.676990 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.678285 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.679123 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.680451 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.680887 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.681034 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.684789 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.684794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.684797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.684830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.688120 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.688847 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.688852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.688856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.689042 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.689443 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.689449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.689461 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.689738 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.689897 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.690543 (+ 646us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.690750 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":318,"threads_started":1}
I20260809 06:22:12.692799 18275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:12.708119 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.693439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:41065 (local address 127.0.0.1:45036)
0809 06:22:12.693624 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.693643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.693761 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.694091 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.694096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.694561 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.694738 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.694747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.696046 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.696053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.697429 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.697446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.698378 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.698384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.698605 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.699356 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.699382 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.701762 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.705472 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.705480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.705485 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.705910 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.706364 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.706367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.706369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.706501 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.707123 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.707127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.707336 (+ 209us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.707515 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.707790 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 06:22:12.708168 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.693654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.3.137.125:41065)
0809 06:22:12.693821 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.693825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.693840 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:22:12.693887 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.693891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.693945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.694029 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.694886 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.695898 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.697563 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.698219 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.698341 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.701956 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.701960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.701964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.701994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.705337 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.706033 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.706041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.706047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.706213 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.706614 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.706619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.706622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.706991 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.707137 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.707618 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.707792 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:22:12.709337 18275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:22:12.710314 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.692380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.3.137.126:43659)
0809 06:22:12.692564 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.692570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.692612 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:22:12.692792 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.692797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.692884 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.692998 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.694594 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.695645 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.697560 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.698210 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.698341 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.703732 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.703739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.703747 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.703821 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.707112 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.707975 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.707982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.707987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.708128 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.708626 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.708631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.708633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.709017 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.709286 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.709668 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.709941 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:22:12.722872 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.710053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43094)
0809 06:22:12.710223 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.710245 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.710357 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.710775 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.710779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.711237 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.711368 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.711376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.712425 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.712430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.713447 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.713459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.714146 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.714152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.714334 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.714881 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.714897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.717109 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.720448 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.720452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.720455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.720801 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.721193 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.721196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.721198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.721323 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.721843 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.721847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.721989 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.722136 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.722439 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 06:22:12.722904 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.710311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.3.137.124:39449)
0809 06:22:12.710491 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.710497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.710512 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:22:12.710555 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.710560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.710627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.710734 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.711528 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.712280 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.713581 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.714006 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.714130 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.717345 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.717353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.717359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.717398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.720328 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.720926 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.720933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.720936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.721079 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.721432 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.721439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.721442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.721724 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.721878 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.722209 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.722594 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":66}
I20260809 06:22:12.727373 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.712273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.3.137.125:41065)
0809 06:22:12.712445 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.712450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.712468 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:22:12.712593 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.712598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.712665 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.712774 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.713938 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.714785 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.716119 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.716593 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.716690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.721427 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.721432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.721434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.721463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.724778 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.725409 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.725416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.725420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.725587 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.725934 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.725940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.725944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.726267 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.726555 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.726761 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.727037 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:22:12.729110 18275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: Bootstrap starting.
I20260809 06:22:12.732337 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:22:12.733140 18275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:12.734427 18275 log.cc:824] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:12.737692 18275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: No bootstrap required, opened a new log
I20260809 06:22:12.749707 18275 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.750283 18275 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:12.750476 18275 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd335c70200249fd89d0181c75abbd6e, State: Initialized, Role: FOLLOWER
I20260809 06:22:12.750432 18119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 4
I20260809 06:22:12.751372 18275 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.759306 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.738613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.3.137.126:43659)
0809 06:22:12.738787 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.738793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.739131 (+ 338us) server_negotiation.cc:407] Connection header received
0809 06:22:12.739441 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.739446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.739534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.739610 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.741944 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.743077 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.745264 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.745940 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.746043 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.750271 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.750280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.750284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.750336 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.754474 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.755569 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.755579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.755586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.755805 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.756346 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.756354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.756360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.756825 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.757129 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.758436 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.758748 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:22:12.760107 18291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:12.760526 18291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:12.761417 18275 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:12.762535 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.739740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.3.137.124:39449)
0809 06:22:12.739947 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.739952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.741262 (+ 1310us) server_negotiation.cc:407] Connection header received
0809 06:22:12.741554 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.741561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.741639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.741714 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.744034 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.745240 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.747396 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.748068 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.748210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.752688 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.752715 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.752722 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.752773 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.758495 (+ 5722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.759706 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.759714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.759718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.759892 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.760646 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.760652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.760658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.761034 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.761293 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.761905 (+ 612us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.762139 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:22:12.767524 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.738208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.3.137.125:41065)
0809 06:22:12.738368 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.738376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.739327 (+ 951us) server_negotiation.cc:407] Connection header received
0809 06:22:12.739974 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.739982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.740076 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.740161 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.742916 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.744031 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.746088 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.746730 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.746822 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.758286 (+ 11464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.758294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.758300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.758341 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.762725 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.764348 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.764356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.764363 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.764562 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.765134 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.765142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.765148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.765618 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.765904 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.766648 (+ 744us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.767021 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:22:12.781708 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.751779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.3.137.124:39449)
0809 06:22:12.763017 (+ 11238us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.763026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.763048 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:22:12.763119 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.763127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.763213 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.764674 (+ 1461us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.764884 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.766400 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.769553 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.770806 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.771015 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.774023 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.774033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.774040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.774086 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.778537 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.779441 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.779449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.779455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.779621 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.780097 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.780100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.780102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.780498 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.780667 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.781065 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.781274 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11044,"thread_start_us":17865,"threads_started":1}
I20260809 06:22:12.781697 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.751037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43110)
0809 06:22:12.751402 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.751420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.751547 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.763434 (+ 11887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.763441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.764135 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.764478 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.764493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.766632 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.766641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.768599 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.768618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.771024 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.771031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.771179 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.771859 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.771876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.773832 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.778768 (+ 4936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.778775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.778781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.779294 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.779806 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.779811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.779814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.779977 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.780618 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.780622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.780797 (+ 175us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.780964 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.781265 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260809 06:22:12.787333 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: Bootstrap starting.
W20260809 06:22:12.788352 18302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:22:12.788529 18302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:22:12.791460 18119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:12.792826 18119 log.cc:824] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:12.783051 18203 master_runner.cc:186] Error getting master registration for 127.3.137.126:43659: OK, instance_id { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" instance_seqno: 1786256531610514 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:22:12.794585 18203 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:22:12.796134 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: No bootstrap required, opened a new log
I20260809 06:22:12.807016 18119 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.807425 18119 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:12.807566 18119 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b63e98307e487ca3c74d812f82926e, State: Initialized, Role: FOLLOWER
I20260809 06:22:12.808269 18119 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.809163 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:12.809540 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:12.810534 18119 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: configured and running, proceeding with master startup.
W20260809 06:22:12.821110 18317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:22:12.821331 18317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:22:12.820664 18044 master_runner.cc:186] Error getting master registration for 127.3.137.125:41065: OK, instance_id { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" instance_seqno: 1786256532119892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:22:12.832321 18044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:22:12.847913 18196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } attempt: 5
W20260809 06:22:12.850551 18066 tablet.cc:2369] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 06:22:12 dist-test-slave-2w3w krb5kdc[18031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:22:12.866320 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.848525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:39449 (local address 127.0.0.1:43116)
0809 06:22:12.848803 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:12.848835 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:12.848957 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:12.849367 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:12.849372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:12.849954 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:12.850163 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.850176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.852753 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.852759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.854161 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:12.854176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.855240 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.855254 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:12.855475 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.856162 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:12.856187 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:12.859061 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:12.863386 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.863391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.863396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.863923 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.864405 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:12.864411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:12.864414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:12.864581 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:12.865311 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:12.865316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:12.865479 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 06:22:12.865640 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 06:22:12.865924 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260809 06:22:12.867439 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:12.848630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.3.137.124:39449)
0809 06:22:12.848806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:12.848813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:12.848911 (+ 98us) server_negotiation.cc:407] Connection header received
0809 06:22:12.849094 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:12.849101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:12.849181 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:12.849295 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:12.851423 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.852557 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.854321 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:12.855075 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:12.855211 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:12.859281 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:12.859289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:12.859294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:12.859332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:12.863188 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.864084 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.864090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.864094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.864266 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:12.864713 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:12.864719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:12.864722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:12.865081 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:12.865223 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:12.866753 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0809 06:22:12.867044 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:22:12.873126 18196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: Bootstrap starting.
I20260809 06:22:12.877173 18196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:12.878547 18196 log.cc:824] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:12.881843 18196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: No bootstrap required, opened a new log
W20260809 06:22:12.893180 18225 tablet.cc:2369] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:12.893055 18196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.893427 18196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:12.893527 18196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1b72bb4e7046648aa394c252fc0291, State: Initialized, Role: FOLLOWER
I20260809 06:22:12.894191 18196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.895231 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:12.895704 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:12.896402 18196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:12.914287 18126 master_runner.cc:263] No leader master found from master 6e1b72bb4e7046648aa394c252fc0291
I20260809 06:22:12.914433 18126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 06:22:12.917622 18333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:22:12.917858 18333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:22:12.990093 18305 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:22:12.990594 18305 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:12.992421 18305 leader_election.cc:288] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e1b72bb4e7046648aa394c252fc0291 (127.3.137.125:41065), bd335c70200249fd89d0181c75abbd6e (127.3.137.124:39449)
I20260809 06:22:12.993683 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0b63e98307e487ca3c74d812f82926e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd335c70200249fd89d0181c75abbd6e" is_pre_election: true
I20260809 06:22:12.994341 18248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b63e98307e487ca3c74d812f82926e in term 0.
I20260809 06:22:12.995213 18061 leader_election.cc:302] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd335c70200249fd89d0181c75abbd6e, e0b63e98307e487ca3c74d812f82926e; no voters:
I20260809 06:22:12.996526 18170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0b63e98307e487ca3c74d812f82926e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1b72bb4e7046648aa394c252fc0291" is_pre_election: true
I20260809 06:22:12.997151 18170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b63e98307e487ca3c74d812f82926e in term 0.
I20260809 06:22:12.998145 18305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:22:12.998308 18305 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:22:12.998404 18305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:13.002090 18305 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:13.007642 18170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0b63e98307e487ca3c74d812f82926e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1b72bb4e7046648aa394c252fc0291"
I20260809 06:22:13.007846 18170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:13.011236 18170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b63e98307e487ca3c74d812f82926e in term 1.
I20260809 06:22:13.011751 18064 leader_election.cc:302] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1b72bb4e7046648aa394c252fc0291, e0b63e98307e487ca3c74d812f82926e; no voters:
I20260809 06:22:13.013887 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0b63e98307e487ca3c74d812f82926e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd335c70200249fd89d0181c75abbd6e"
I20260809 06:22:13.014112 18248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:13.018739 18248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b63e98307e487ca3c74d812f82926e in term 1.
I20260809 06:22:13.019601 18305 leader_election.cc:288] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [CANDIDATE]: Term 1 election: Requested vote from peers 6e1b72bb4e7046648aa394c252fc0291 (127.3.137.125:41065), bd335c70200249fd89d0181c75abbd6e (127.3.137.124:39449)
I20260809 06:22:13.020102 18305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:13.020536 18305 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [term 1 LEADER]: Becoming Leader. State: Replica: e0b63e98307e487ca3c74d812f82926e, State: Running, Role: LEADER
I20260809 06:22:13.021339 18305 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } }
I20260809 06:22:13.037597 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b63e98307e487ca3c74d812f82926e. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.038004 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:13.055428 18337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:13.057925 18337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:13.078331 18248 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [term 1 FOLLOWER]: Refusing update from remote peer e0b63e98307e487ca3c74d812f82926e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:13.079576 18305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:13.087397 18170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [term 1 FOLLOWER]: Refusing update from remote peer e0b63e98307e487ca3c74d812f82926e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:13.088397 18305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:22:13.109232 18147 tablet.cc:2369] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:13.124253 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b63e98307e487ca3c74d812f82926e. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.124635 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:13.130188 18338 mvcc.cc:205] Tried to move back new op lower bound from 7316506759475720192 to 7316506759265996800. Current Snapshot: MvccSnapshot[applied={T|T < 7316506759475720192}]
I20260809 06:22:13.134881 18291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b63e98307e487ca3c74d812f82926e. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.136750 18291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:13.139446 18320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.139683 18320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:13.141779 18339 mvcc.cc:205] Tried to move back new op lower bound from 7316506759475720192 to 7316506759265996800. Current Snapshot: MvccSnapshot[applied={T|T < 7316506759475720192}]
I20260809 06:22:13.147467 18291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.147696 18291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:22:13.149974 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.150249 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:13.150540 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0b63e98307e487ca3c74d812f82926e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b63e98307e487ca3c74d812f82926e" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43659 } } peers { permanent_uuid: "6e1b72bb4e7046648aa394c252fc0291" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 41065 } } peers { permanent_uuid: "bd335c70200249fd89d0181c75abbd6e" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 39449 } } }
I20260809 06:22:13.150779 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:13.151507 18337 catalog_manager.cc:1347] Generated new cluster ID: a1f7d98d933d413f8e60ead3a1ddf3f4
I20260809 06:22:13.151607 18337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:13.195763 18337 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:22:13.198000 18337 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:13.237278 18337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0b63e98307e487ca3c74d812f82926e: Generated new TSK 0
I20260809 06:22:13.238567 18337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:22:13.790688 18302 catalog_manager.cc:1259] Loaded cluster ID: a1f7d98d933d413f8e60ead3a1ddf3f4
I20260809 06:22:13.790793 18302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: loading cluster ID for follower catalog manager: success
I20260809 06:22:13.793496 18302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: acquiring CA information for follower catalog manager: success
I20260809 06:22:13.795393 18302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd335c70200249fd89d0181c75abbd6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:22:13.920011 18333 catalog_manager.cc:1259] Loaded cluster ID: a1f7d98d933d413f8e60ead3a1ddf3f4
I20260809 06:22:13.920112 18333 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: loading cluster ID for follower catalog manager: success
I20260809 06:22:13.922459 18333 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: acquiring CA information for follower catalog manager: success
I20260809 06:22:13.924177 18333 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e1b72bb4e7046648aa394c252fc0291: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:22:14.367857 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18044
I20260809 06:22:14.369467 18114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:14.479278 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18044
I20260809 06:22:14.502890 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18126
I20260809 06:22:14.504055 18191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:14.609694 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18126
I20260809 06:22:14.631628 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18203
I20260809 06:22:14.632824 18270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:14.740907 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18203
2026-08-09T06:22:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4991 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:22:14 dist-test-slave-2w3w krb5kdc[18357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:22:14 dist-test-slave-2w3w krb5kdc[18357](info): set up 2 sockets
Aug 09 06:22:14 dist-test-slave-2w3w krb5kdc[18357](info): commencing operation
krb5kdc: starting...
W20260809 06:22:15.909112 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:22:16.183113 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46297
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:46297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:16.524971 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:16.525244 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:16.525332 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:16.532342 18372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:16.532445 18372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:16.532490 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:16.532546 18372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:16.532588 18372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:16.542232 18372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:46297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:16.543416 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:16.544627 18372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:16.556337 18380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:16.556311 18377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:16.556532 18378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:16.557801 18372 server_base.cc:1034] running on GCE node
I20260809 06:22:16.558573 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:16.560133 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:16.561395 18372 hybrid_clock.cc:648] HybridClock initialized: now 1786256536561334 us; error 52 us; skew 500 ppm
I20260809 06:22:16.561764 18372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256536, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:16.564862 18372 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:16.566356 18372 webserver.cc:460] Webserver started at http://127.3.137.126:37863/ using document root <none> and password file <none>
I20260809 06:22:16.567034 18372 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:16.567150 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:16.567428 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:16.570320 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "4b98a67002354a1c8e57b8e370e5c01a"
format_stamp: "Formatted at 2026-08-09 06:22:16 on dist-test-slave-2w3w"
server_key: "1c9d240daa1f0aaa800be00510332307"
server_key_iv: "340066381bb4f26da963d11224287a93"
server_key_version: "encryptionkey@0"
I20260809 06:22:16.571095 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "4b98a67002354a1c8e57b8e370e5c01a"
format_stamp: "Formatted at 2026-08-09 06:22:16 on dist-test-slave-2w3w"
server_key: "1c9d240daa1f0aaa800be00510332307"
server_key_iv: "340066381bb4f26da963d11224287a93"
server_key_version: "encryptionkey@0"
I20260809 06:22:16.576176 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:22:16.579564 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:16.580899 18372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:22:16.581068 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "4b98a67002354a1c8e57b8e370e5c01a"
format_stamp: "Formatted at 2026-08-09 06:22:16 on dist-test-slave-2w3w"
server_key: "1c9d240daa1f0aaa800be00510332307"
server_key_iv: "340066381bb4f26da963d11224287a93"
server_key_version: "encryptionkey@0"
I20260809 06:22:16.581280 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:16.600523 18372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:16.604585 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:16.604885 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:16.624320 18372 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:46297
I20260809 06:22:16.624353 18443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:46297 every 8 connection(s)
I20260809 06:22:16.625641 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:16.627211 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18372
I20260809 06:22:16.627494 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:22:16.628403 3621 external_mini_cluster.cc:1428] Setting key 36b70e2780352080aa21ca2f3a19092d
I20260809 06:22:16.630440 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:16.643831 18444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a: Bootstrap starting.
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:16.647383 18444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:16.648406 18444 log.cc:824] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:16.651770 18444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a: No bootstrap required, opened a new log
I20260809 06:22:16.652838 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:16.630264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.3.137.126:46297)
0809 06:22:16.631054 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:16.631068 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:16.631116 (+ 48us) server_negotiation.cc:407] Connection header received
0809 06:22:16.633416 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:16.633442 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:16.633934 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:16.634364 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:16.635845 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:16.637928 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:16.639256 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:16.640340 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:16.640543 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:16.645300 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:16.645308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:16.645323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:16.645361 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:16.648431 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:16.649346 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:16.649357 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:16.649367 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:16.649586 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:16.650060 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:16.650063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:16.650065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:16.650565 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:16.650815 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:16.651360 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 06:22:16.651678 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":331,"threads_started":1}
I20260809 06:22:16.656715 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:22:16.659782 18444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } }
I20260809 06:22:16.660228 18444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:16.660336 18444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b98a67002354a1c8e57b8e370e5c01a, State: Initialized, Role: FOLLOWER
I20260809 06:22:16.660868 18444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } }
I20260809 06:22:16.661108 18444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:16.661250 18444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:16.661427 18444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:16.663607 18444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } }
I20260809 06:22:16.664093 18444 leader_election.cc:302] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b98a67002354a1c8e57b8e370e5c01a; no voters:
I20260809 06:22:16.664659 18444 leader_election.cc:288] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:22:16.664791 18450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:16.665516 18450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [term 1 LEADER]: Becoming Leader. State: Replica: 4b98a67002354a1c8e57b8e370e5c01a, State: Running, Role: LEADER
I20260809 06:22:16.666105 18450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } }
I20260809 06:22:16.666640 18444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:16.668905 18452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b98a67002354a1c8e57b8e370e5c01a. Latest consensus state: current_term: 1 leader_uuid: "4b98a67002354a1c8e57b8e370e5c01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } } }
I20260809 06:22:16.669401 18452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:16.668923 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b98a67002354a1c8e57b8e370e5c01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98a67002354a1c8e57b8e370e5c01a" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46297 } } }
I20260809 06:22:16.669781 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:16.675776 18456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:16.681576 18456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:16.690658 18456 catalog_manager.cc:1347] Generated new cluster ID: f5af2254fbc74cd3ae967e01500b3007
I20260809 06:22:16.690783 18456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:16.720497 18456 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:22:16.721899 18456 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:16.727676 18456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b98a67002354a1c8e57b8e370e5c01a: Generated new TSK 0
I20260809 06:22:16.728451 18456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:22:16.769573 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18372
I20260809 06:22:16.770912 18439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:16.864068 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18372
2026-08-09T06:22:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2119 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18474](info): set up 2 sockets
Aug 09 06:22:16 dist-test-slave-2w3w krb5kdc[18474](info): commencing operation
krb5kdc: starting...
W20260809 06:22:18.021226 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:22:18 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:22:18.286932 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:18.612198 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:18.612453 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:18.612601 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:18.619549 18487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:18.619637 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:18.619719 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:18.619837 18487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:18.619920 18487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:18.629171 18487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:42343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:18.630368 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:18.631645 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:18.643191 18493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:18.643229 18495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:18.643142 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:18.644330 18487 server_base.cc:1034] running on GCE node
I20260809 06:22:18.645179 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:18.646461 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:18.647665 18487 hybrid_clock.cc:648] HybridClock initialized: now 1786256538647595 us; error 63 us; skew 500 ppm
I20260809 06:22:18.648029 18487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:18 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:18.654047 18487 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:18.655632 18487 webserver.cc:460] Webserver started at http://127.3.137.126:36655/ using document root <none> and password file <none>
I20260809 06:22:18.656298 18487 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:18.656441 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:18.656733 18487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:18.659684 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "9f1d5e9706254fb89e1806365522e518"
format_stamp: "Formatted at 2026-08-09 06:22:18 on dist-test-slave-2w3w"
server_key: "97f6b05a5aeaccee9a90be79f73d4adc"
server_key_iv: "c1514b7741b1e6eb4e6ae099e59c1ee1"
server_key_version: "encryptionkey@0"
I20260809 06:22:18.660432 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "9f1d5e9706254fb89e1806365522e518"
format_stamp: "Formatted at 2026-08-09 06:22:18 on dist-test-slave-2w3w"
server_key: "97f6b05a5aeaccee9a90be79f73d4adc"
server_key_iv: "c1514b7741b1e6eb4e6ae099e59c1ee1"
server_key_version: "encryptionkey@0"
I20260809 06:22:18.665383 18487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 06:22:18.668584 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:18.669772 18487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 06:22:18.669960 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "9f1d5e9706254fb89e1806365522e518"
format_stamp: "Formatted at 2026-08-09 06:22:18 on dist-test-slave-2w3w"
server_key: "97f6b05a5aeaccee9a90be79f73d4adc"
server_key_iv: "c1514b7741b1e6eb4e6ae099e59c1ee1"
server_key_version: "encryptionkey@0"
I20260809 06:22:18.670149 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:18.688362 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:18.692667 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:18.692952 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:18.711776 18487 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:42343
I20260809 06:22:18.711781 18559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:42343 every 8 connection(s)
I20260809 06:22:18.713253 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:18.716871 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:18.722043 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18487
I20260809 06:22:18.722306 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:22:18.723111 3621 external_mini_cluster.cc:1428] Setting key bddc9a7070c0e6c4b0ba9453dd1760f6
I20260809 06:22:18.728678 18560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap starting.
I20260809 06:22:18.732426 18560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:18.733654 18560 log.cc:824] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:18.737111 18560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: No bootstrap required, opened a new log
Aug 09 06:22:18 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:18.745280 18560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:18.745654 18560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:18.745769 18560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Initialized, Role: FOLLOWER
I20260809 06:22:18.746430 18560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:18.746726 18560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:18.746882 18560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:18.747115 18560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:18.747728 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:18.725413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.3.137.126:42343)
0809 06:22:18.726283 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:18.726299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:18.726342 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:22:18.728149 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:18.728172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:18.728557 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:18.728931 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:18.730790 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:18.732116 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:18.734033 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:18.735201 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:18.735333 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:18.740172 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:18.740180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:18.740192 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:18.740230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:18.743487 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:18.744190 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:18.744197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:18.744206 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:18.744377 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:18.744785 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:18.744795 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:18.744797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:18.745394 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:18.745705 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:18.746166 (+ 461us) server_negotiation.cc:299] Negotiation successful
0809 06:22:18.746582 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":393,"threads_started":1}
I20260809 06:22:18.749976 18560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:18.750603 18560 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1d5e9706254fb89e1806365522e518; no voters:
I20260809 06:22:18.751256 18560 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:22:18.751395 18565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:18.752125 18565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Running, Role: LEADER
I20260809 06:22:18.752615 18565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:18.753268 18560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:18.755585 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1d5e9706254fb89e1806365522e518. Latest consensus state: current_term: 1 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:18.755954 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:18.757025 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:18.757505 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
W20260809 06:22:18.767022 18582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:22:18.767127 18582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:22:18.767274 18573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:18.768741 18573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:18.776204 18573 catalog_manager.cc:1347] Generated new cluster ID: 1cd631c1964b4ea3b263e86da66ac75d
I20260809 06:22:18.776321 18573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:18.804090 18573 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:22:18.806025 18573 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:18.819380 18573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Generated new TSK 0
I20260809 06:22:18.820384 18573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:22:18.890890 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:19.224184 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:19.224428 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:19.224623 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:19.231482 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:19.231575 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:19.231771 18587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:19.241223 18587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:19.242443 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:19.243714 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:19.256963 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:19.257009 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:19.256996 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:19.258086 18587 server_base.cc:1034] running on GCE node
I20260809 06:22:19.258667 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:19.259658 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:19.260852 18587 hybrid_clock.cc:648] HybridClock initialized: now 1786256539260807 us; error 44 us; skew 500 ppm
I20260809 06:22:19.261168 18587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:19 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256539, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:19.264204 18587 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:19.265539 18587 webserver.cc:460] Webserver started at http://127.3.137.65:34431/ using document root <none> and password file <none>
I20260809 06:22:19.266147 18587 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:19.266243 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:19.266479 18587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:19.269238 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "7967018d49534e3f8efee5d807dc9b66"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "1116aacacd3b777c78b77d2343f42405"
server_key_iv: "38617ee99b6244d3ad60bbec658a96b6"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.269912 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "7967018d49534e3f8efee5d807dc9b66"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "1116aacacd3b777c78b77d2343f42405"
server_key_iv: "38617ee99b6244d3ad60bbec658a96b6"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.274695 18587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 06:22:19.277967 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.279326 18587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:19.279489 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "7967018d49534e3f8efee5d807dc9b66"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "1116aacacd3b777c78b77d2343f42405"
server_key_iv: "38617ee99b6244d3ad60bbec658a96b6"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.279687 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:19.306288 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:19.310155 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:19.310448 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:19.311372 18587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:19.312637 18587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:19.312711 18587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.312892 18587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:19.312954 18587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.346247 18587 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40059
I20260809 06:22:19.346316 18720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40059 every 8 connection(s)
I20260809 06:22:19.347545 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:19.348908 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18587
I20260809 06:22:19.349164 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:22:19.350021 3621 external_mini_cluster.cc:1428] Setting key 3b3c80e0e7115d56529d570969de0e2f
Aug 09 06:22:19 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256539, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:19.372561 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:19.350375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:56151 (local address 127.3.137.126:42343)
0809 06:22:19.350859 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:19.350866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:19.352026 (+ 1160us) server_negotiation.cc:407] Connection header received
0809 06:22:19.354817 (+ 2791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:19.354821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:19.354897 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:19.355064 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:19.357080 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.357809 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.360328 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.360736 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.360828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:19.365396 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:19.365401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:19.365404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:19.365435 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:19.368913 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:19.369636 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:19.369644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:19.369650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:19.369823 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:19.370243 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:19.370251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:19.370257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:19.370631 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:19.370786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:19.371692 (+ 906us) server_negotiation.cc:299] Negotiation successful
0809 06:22:19.371957 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260809 06:22:19.373167 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:19.350985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:56151)
0809 06:22:19.351731 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:19.351767 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:19.354449 (+ 2682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:19.355292 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:19.355305 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:19.356077 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:19.356880 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:19.356908 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.357969 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.357977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:19.360186 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:19.360205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.360946 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.360964 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:19.361293 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:19.362138 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:19.362176 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:19.365173 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:19.369064 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:19.369070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:19.369081 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:19.369494 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:19.369933 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:19.369936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:19.369939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:19.370111 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:19.370751 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:19.370757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:19.371277 (+ 520us) client_negotiation.cc:769] Sending connection context
0809 06:22:19.371527 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:22:19.371894 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
I20260809 06:22:19.374876 18721 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:19.375219 18721 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:19.375932 18721 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:19.378297 18521 ts_manager.cc:194] Registered new tserver with Master: 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:19.380575 18521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:56151
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:22:19.414772 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:19.736480 18727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:19.736738 18727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:19.736948 18727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:19.744099 18727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:19.744220 18727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:19.744405 18727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:19.753871 18727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:19.755211 18727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:19.756466 18727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:19.769173 18732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:19.769147 18733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:19.769232 18735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:19.770103 18727 server_base.cc:1034] running on GCE node
I20260809 06:22:19.770711 18727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:19.771648 18727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:19.772876 18727 hybrid_clock.cc:648] HybridClock initialized: now 1786256539772836 us; error 60 us; skew 500 ppm
I20260809 06:22:19.773213 18727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:19 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256539, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:19.776464 18727 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:19.777750 18727 webserver.cc:460] Webserver started at http://127.3.137.66:42959/ using document root <none> and password file <none>
I20260809 06:22:19.778365 18727 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:19.778462 18727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:19.778695 18727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:19.781598 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "fea177e632284c95850ac019913d1bc6"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "9b2f3fbe9edfbcf9a5bca508698014ff"
server_key_iv: "e4ecc55f13eb5280dfb646a1bf26159e"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.782313 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "fea177e632284c95850ac019913d1bc6"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "9b2f3fbe9edfbcf9a5bca508698014ff"
server_key_iv: "e4ecc55f13eb5280dfb646a1bf26159e"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.787536 18727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:22:19.790685 18743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.792024 18727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:19.792237 18727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "fea177e632284c95850ac019913d1bc6"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "9b2f3fbe9edfbcf9a5bca508698014ff"
server_key_iv: "e4ecc55f13eb5280dfb646a1bf26159e"
server_key_version: "encryptionkey@0"
I20260809 06:22:19.792431 18727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:19.815371 18727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:19.819111 18727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:19.819427 18727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:19.820417 18727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:19.821677 18727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:19.821751 18727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.821832 18727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:19.821877 18727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:19.855664 18727 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40135
I20260809 06:22:19.855749 18861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40135 every 8 connection(s)
I20260809 06:22:19.856917 18727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:19.860245 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18727
I20260809 06:22:19.860497 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:22:19.861277 3621 external_mini_cluster.cc:1428] Setting key b1051594b4f596d38f968f2243aa3ed5
Aug 09 06:22:19 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256539, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:19.879798 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:19.859452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:35855 (local address 127.3.137.126:42343)
0809 06:22:19.859631 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:19.859635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:19.862738 (+ 3103us) server_negotiation.cc:407] Connection header received
0809 06:22:19.864622 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:19.864629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:19.864719 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:19.864834 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:19.866214 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.867409 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.868934 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.869664 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.869798 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:19.873628 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:19.873637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:19.873645 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:19.873690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:19.876591 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:19.877500 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:19.877504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:19.877506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:19.877619 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:19.878093 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:19.878096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:19.878098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:19.878364 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:19.878476 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:19.879381 (+ 905us) server_negotiation.cc:299] Negotiation successful
0809 06:22:19.879560 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:22:19.881255 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:19.859998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:35855)
0809 06:22:19.862561 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:19.862613 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:19.864342 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:19.865016 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:19.865023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:19.865521 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:19.866041 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:19.866056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.867568 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.867572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:19.868785 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:19.868796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:19.869856 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:19.869860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:19.870040 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:19.870766 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:19.870787 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:19.873283 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:19.876725 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:19.876738 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:19.876755 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:19.877387 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:19.877724 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:19.877731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:19.877737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:19.877987 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:19.878515 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:19.878526 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:19.878986 (+ 460us) client_negotiation.cc:769] Sending connection context
0809 06:22:19.879288 (+ 302us) client_negotiation.cc:240] Negotiation successful
0809 06:22:19.879867 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2280,"thread_start_us":2039,"threads_started":1}
I20260809 06:22:19.882727 18862 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:19.883118 18862 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:19.883817 18862 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:19.885311 18521 ts_manager.cc:194] Registered new tserver with Master: fea177e632284c95850ac019913d1bc6 (127.3.137.66:40135)
I20260809 06:22:19.886107 18521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:35855
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:22:19.925989 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:20.251339 18867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:20.251639 18867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:20.251812 18867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:20.259419 18867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:20.259565 18867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:20.259732 18867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:20.269205 18867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:20.270550 18867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:20.272011 18867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:20.285880 18872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:20.285804 18873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:20.285841 18875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:20.286530 18867 server_base.cc:1034] running on GCE node
I20260809 06:22:20.287199 18867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:20.288163 18867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:20.289387 18867 hybrid_clock.cc:648] HybridClock initialized: now 1786256540289358 us; error 60 us; skew 500 ppm
I20260809 06:22:20.289700 18867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:20 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256540, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:20.292872 18867 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:20.294291 18867 webserver.cc:460] Webserver started at http://127.3.137.67:40961/ using document root <none> and password file <none>
I20260809 06:22:20.295018 18867 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:20.295168 18867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:20.295465 18867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:20.298254 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "064718086ad5496085fc62f5697360b4"
format_stamp: "Formatted at 2026-08-09 06:22:20 on dist-test-slave-2w3w"
server_key: "579fffd5aae6cb479157c0c3093aebe6"
server_key_iv: "7394709dcaac20332ba983dc45b64d39"
server_key_version: "encryptionkey@0"
I20260809 06:22:20.299073 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "064718086ad5496085fc62f5697360b4"
format_stamp: "Formatted at 2026-08-09 06:22:20 on dist-test-slave-2w3w"
server_key: "579fffd5aae6cb479157c0c3093aebe6"
server_key_iv: "7394709dcaac20332ba983dc45b64d39"
server_key_version: "encryptionkey@0"
I20260809 06:22:20.303934 18867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 06:22:20.307180 18882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:20.308387 18867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:22:20.308566 18867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "064718086ad5496085fc62f5697360b4"
format_stamp: "Formatted at 2026-08-09 06:22:20 on dist-test-slave-2w3w"
server_key: "579fffd5aae6cb479157c0c3093aebe6"
server_key_iv: "7394709dcaac20332ba983dc45b64d39"
server_key_version: "encryptionkey@0"
I20260809 06:22:20.308764 18867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:20.322052 18867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:20.325873 18867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:20.326159 18867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:20.327095 18867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:20.328344 18867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:20.328449 18867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:20.328603 18867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:20.328683 18867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:20.365588 18867 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33963
I20260809 06:22:20.365635 19007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33963 every 8 connection(s)
I20260809 06:22:20.367151 18867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:20.370342 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 18867
I20260809 06:22:20.370538 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:22:20.371101 3621 external_mini_cluster.cc:1428] Setting key 7db5d5ff80cce16dbb7deae92310c1cc
Aug 09 06:22:20 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256540, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:20.383726 18721 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:20.387629 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.369966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:57769 (local address 127.3.137.126:42343)
0809 06:22:20.370198 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.370205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.371435 (+ 1230us) server_negotiation.cc:407] Connection header received
0809 06:22:20.373453 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.373459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.373529 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.373610 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:20.375229 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.375983 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.377458 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.377837 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.377928 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.381111 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:20.381116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:20.381118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:20.381149 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:20.384102 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.384980 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.384985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.384988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.385161 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.385779 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.385784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.385787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.386140 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:20.386302 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.387074 (+ 772us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.387288 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:22:20.388532 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.370386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:57769)
0809 06:22:20.371262 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.371307 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.373101 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.373876 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.373888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.374459 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:20.375082 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.375101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.376110 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.376114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.377336 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.377348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.378016 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.378021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.378199 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.378776 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:20.378797 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:20.380919 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:20.384258 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:20.384264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:20.384274 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:20.384838 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:20.385373 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:20.385376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:20.385378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:20.385655 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:20.386277 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:20.386285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:20.386666 (+ 381us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.386928 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.387432 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":363,"threads_started":1}
I20260809 06:22:20.389842 19009 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:20.390127 19009 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:20.390785 19009 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:20.392158 18520 ts_manager.cc:194] Registered new tserver with Master: 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963)
I20260809 06:22:20.392992 18520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:57769
I20260809 06:22:20.398533 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:22:20.415782 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.402920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.3.137.126:42343)
0809 06:22:20.403133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.403141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.403553 (+ 412us) server_negotiation.cc:407] Connection header received
0809 06:22:20.403755 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.403761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.403828 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.403929 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:20.405328 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.406065 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.407536 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.407973 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.408124 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.410511 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:20.410516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:20.410519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:20.410549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:20.413060 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.413716 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.413723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.413726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.413867 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.414255 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.414263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.414266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.414579 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:20.414856 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.415223 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.415441 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:22:20.419658 18520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:22:20.426590 18520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:22:20.461688 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.438133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.3.137.66:40135)
0809 06:22:20.449047 (+ 10914us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.449058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.453133 (+ 4075us) server_negotiation.cc:407] Connection header received
0809 06:22:20.453253 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.453260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.453526 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.453761 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:20.454783 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.456419 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.459114 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.460510 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.460652 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.460854 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.461173 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.461351 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10718,"thread_start_us":13745,"threads_started":1}
I20260809 06:22:20.461774 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.437949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40135 (local address 127.0.0.1:41224)
0809 06:22:20.452626 (+ 14677us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.452676 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.452883 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.454109 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.454120 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.454155 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:20.454554 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.454567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.456597 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.456604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.458878 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.458895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.460712 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.460723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.460935 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.460989 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.461117 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.461301 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14411,"mutex_wait_us":3,"thread_start_us":14568,"threads_started":1}
I20260809 06:22:20.463368 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.438154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33963 (local address 127.0.0.1:44900)
0809 06:22:20.454058 (+ 15904us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.454076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.454183 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.455242 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.455247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.455271 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:20.455783 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.455797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.457981 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.457989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.460282 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.460302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.462552 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.462558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.462724 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.462755 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.462856 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.463009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15751,"thread_start_us":15631,"threads_started":1}
I20260809 06:22:20.464265 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.438522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.3.137.67:33963)
0809 06:22:20.452045 (+ 13523us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.452053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.454322 (+ 2269us) server_negotiation.cc:407] Connection header received
0809 06:22:20.454496 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.454510 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.454823 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.455051 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:20.456163 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.457800 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.460472 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.462017 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.462136 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.462404 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.463737 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.463894 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13315,"thread_start_us":13108,"threads_started":1}
I20260809 06:22:20.464715 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.455275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36360)
0809 06:22:20.455900 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.455918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.456106 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.456977 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.456981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.457009 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:20.457717 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.457730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.459731 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.459736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.461914 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.461930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.463980 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.463985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.464097 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.464117 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.464217 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.464334 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":154,"thread_start_us":327,"threads_started":1}
I20260809 06:22:20.464753 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.454598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.3.137.65:40059)
0809 06:22:20.455271 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.455278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.456013 (+ 735us) server_negotiation.cc:407] Connection header received
0809 06:22:20.456289 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.456297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.456532 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.456693 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:20.457884 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.459548 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.462103 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.463576 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.463682 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.463836 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.464269 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.464418 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":352,"threads_started":1}
I20260809 06:22:20.466013 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d (DEFAULT_TABLE table=test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:20.467600 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c5fbb82ff14ea08107ea7b237be35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:20.467950 18935 tablet_service.cc:1467] Processing CreateTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d (DEFAULT_TABLE table=test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:20.469031 18935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c5fbb82ff14ea08107ea7b237be35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:20.468590 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d (DEFAULT_TABLE table=test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:20.469924 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c5fbb82ff14ea08107ea7b237be35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:20.483405 19029 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Bootstrap starting.
I20260809 06:22:20.485316 19030 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Bootstrap starting.
I20260809 06:22:20.486141 19028 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Bootstrap starting.
I20260809 06:22:20.488960 19030 tablet_bootstrap.cc:654] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:20.489286 19028 tablet_bootstrap.cc:654] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:20.490077 19030 log.cc:824] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:20.490170 19028 log.cc:824] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:20.492848 19030 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: No bootstrap required, opened a new log
I20260809 06:22:20.492842 19029 tablet_bootstrap.cc:654] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:20.493194 19030 ts_tablet_manager.cc:1397] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260809 06:22:20.494087 19029 log.cc:824] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:20.495414 19028 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: No bootstrap required, opened a new log
I20260809 06:22:20.495683 19028 ts_tablet_manager.cc:1397] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260809 06:22:20.497318 19029 tablet_bootstrap.cc:492] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: No bootstrap required, opened a new log
I20260809 06:22:20.497562 19029 ts_tablet_manager.cc:1397] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260809 06:22:20.501698 19030 raft_consensus.cc:348] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.502120 19030 raft_consensus.cc:374] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:20.502242 19030 raft_consensus.cc:729] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7967018d49534e3f8efee5d807dc9b66, State: Initialized, Role: FOLLOWER
I20260809 06:22:20.502874 19030 consensus_queue.cc:260] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.505373 19030 ts_tablet_manager.cc:1428] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260809 06:22:20.506908 19028 raft_consensus.cc:348] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.507335 19028 raft_consensus.cc:374] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:20.507467 19028 raft_consensus.cc:729] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064718086ad5496085fc62f5697360b4, State: Initialized, Role: FOLLOWER
I20260809 06:22:20.508044 19028 consensus_queue.cc:260] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.509692 19009 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:20.509613 19029 raft_consensus.cc:348] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.510027 19029 raft_consensus.cc:374] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:20.510126 19029 raft_consensus.cc:729] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea177e632284c95850ac019913d1bc6, State: Initialized, Role: FOLLOWER
I20260809 06:22:20.510311 19028 ts_tablet_manager.cc:1428] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260809 06:22:20.510672 19029 consensus_queue.cc:260] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.511942 18862 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:20.512445 19029 ts_tablet_manager.cc:1428] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
W20260809 06:22:20.600032 18722 tablet.cc:2369] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:22:20.608817 18863 tablet.cc:2369] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:22:20.619321 19010 tablet.cc:2369] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:20.652758 19036 raft_consensus.cc:482] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:22:20.653035 19036 raft_consensus.cc:504] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.655025 19036 leader_election.cc:288] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963), 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:20.661634 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.655452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:59927 (local address 127.3.137.67:33963)
0809 06:22:20.655637 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.655642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.656065 (+ 423us) server_negotiation.cc:407] Connection header received
0809 06:22:20.656283 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.656288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.656370 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.656481 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:20.657121 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.657901 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.659781 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.660632 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.660823 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.660900 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.661235 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.661383 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:22:20.661662 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.655273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33963 (local address 127.3.137.66:59927)
0809 06:22:20.655951 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.655978 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.656121 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.656551 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.656557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.656601 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:20.656964 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.656973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.658033 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.658042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.659616 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.659627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.660844 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.660850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.661013 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.661037 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.661170 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.661313 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":3,"thread_start_us":350,"threads_started":1}
I20260809 06:22:20.662460 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.656072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.3.137.66:49183)
0809 06:22:20.656738 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:20.656772 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:20.656955 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:20.657307 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:20.657312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:20.657338 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:20.657705 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.657712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.658757 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.658768 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.660311 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:20.660322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.661729 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.661738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:20.661920 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.661966 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 06:22:20.662099 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 06:22:20.662202 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":1,"thread_start_us":377,"threads_started":1}
I20260809 06:22:20.662562 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.656235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:49183 (local address 127.3.137.65:40059)
0809 06:22:20.656433 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.656442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.656890 (+ 448us) server_negotiation.cc:407] Connection header received
0809 06:22:20.657098 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.657103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.657172 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.657284 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:20.657873 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.658623 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.660465 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.661505 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.661667 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.661744 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.662161 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.662303 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:22:20.663033 18956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c5fbb82ff14ea08107ea7b237be35d" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064718086ad5496085fc62f5697360b4" is_pre_election: true
I20260809 06:22:20.663551 18956 raft_consensus.cc:2454] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 0.
I20260809 06:22:20.663549 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c5fbb82ff14ea08107ea7b237be35d" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7967018d49534e3f8efee5d807dc9b66" is_pre_election: true
I20260809 06:22:20.664090 18673 raft_consensus.cc:2454] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 0.
I20260809 06:22:20.664177 18747 leader_election.cc:302] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064718086ad5496085fc62f5697360b4, fea177e632284c95850ac019913d1bc6; no voters:
I20260809 06:22:20.664562 19036 raft_consensus.cc:2781] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:22:20.664729 19036 raft_consensus.cc:482] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:22:20.664841 19036 raft_consensus.cc:3037] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:20.667016 19036 raft_consensus.cc:504] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.667622 19036 leader_election.cc:288] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963), 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:20.668048 18956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c5fbb82ff14ea08107ea7b237be35d" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064718086ad5496085fc62f5697360b4"
I20260809 06:22:20.668111 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c5fbb82ff14ea08107ea7b237be35d" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7967018d49534e3f8efee5d807dc9b66"
I20260809 06:22:20.668290 18673 raft_consensus.cc:3037] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:20.668270 18956 raft_consensus.cc:3037] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:20.671547 18956 raft_consensus.cc:2454] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 1.
I20260809 06:22:20.671536 18673 raft_consensus.cc:2454] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 1.
I20260809 06:22:20.671941 18747 leader_election.cc:302] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064718086ad5496085fc62f5697360b4, fea177e632284c95850ac019913d1bc6; no voters:
I20260809 06:22:20.672222 19036 raft_consensus.cc:2781] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:20.672669 19036 raft_consensus.cc:686] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 1 LEADER]: Becoming Leader. State: Replica: fea177e632284c95850ac019913d1bc6, State: Running, Role: LEADER
I20260809 06:22:20.673180 19036 consensus_queue.cc:237] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:20.678007 18521 catalog_manager.cc:5495] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fea177e632284c95850ac019913d1bc6 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "fea177e632284c95850ac019913d1bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20260809 06:22:20.702523 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.696430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.3.137.66:40135)
0809 06:22:20.696610 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.696614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.696755 (+ 141us) server_negotiation.cc:407] Connection header received
0809 06:22:20.696924 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.696927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.697021 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.697146 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:22:20.697479 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.698185 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.699864 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.700432 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.700572 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.701150 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:22:20.701768 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:22:20.701927 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.702021 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.702102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:22:20.710675 18673 raft_consensus.cc:1261] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Refusing update from remote peer fea177e632284c95850ac019913d1bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:20.710675 18956 raft_consensus.cc:1261] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Refusing update from remote peer fea177e632284c95850ac019913d1bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:20.711777 19040 consensus_queue.cc:1035] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:20.712234 19036 consensus_queue.cc:1035] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:20.729779 19044 mvcc.cc:205] Tried to move back new op lower bound from 7316506790740951040 to 7316506790601740288. Current Snapshot: MvccSnapshot[applied={T|T < 7316506790740951040}]
I20260809 06:22:20.746742 18520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:20.747047 18520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:20.750423 18520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Sending DeleteTablet for 3 replicas of tablet 26c5fbb82ff14ea08107ea7b237be35d
I20260809 06:22:20.751533 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:20 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:41224
I20260809 06:22:20.751770 18651 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:20 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:36360
I20260809 06:22:20.752185 18935 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c5fbb82ff14ea08107ea7b237be35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:20 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:44900
I20260809 06:22:20.752645 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18587
I20260809 06:22:20.762530 19054 tablet_replica.cc:330] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: stopping tablet replica
I20260809 06:22:20.763818 19054 raft_consensus.cc:2229] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:20.765158 19054 raft_consensus.cc:2258] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:20.765681 19053 tablet_replica.cc:330] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: stopping tablet replica
I20260809 06:22:20.765934 19055 tablet_replica.cc:330] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: stopping tablet replica
I20260809 06:22:20.766083 19053 raft_consensus.cc:2229] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:22:20.766709 19055 raft_consensus.cc:2229] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:20.766762 19053 raft_consensus.cc:2258] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:20.767359 19055 raft_consensus.cc:2258] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:20.768666 19054 ts_tablet_manager.cc:1905] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:20.769254 19053 ts_tablet_manager.cc:1905] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:20.769272 19055 ts_tablet_manager.cc:1905] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:20.776590 19055 ts_tablet_manager.cc:1918] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:20.776813 19055 log.cc:1196] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/26c5fbb82ff14ea08107ea7b237be35d
I20260809 06:22:20.777062 19053 ts_tablet_manager.cc:1918] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:20.777233 19053 log.cc:1196] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/26c5fbb82ff14ea08107ea7b237be35d
I20260809 06:22:20.777678 19055 ts_tablet_manager.cc:1939] T 26c5fbb82ff14ea08107ea7b237be35d P 064718086ad5496085fc62f5697360b4: Deleting consensus metadata
I20260809 06:22:20.777825 19053 ts_tablet_manager.cc:1939] T 26c5fbb82ff14ea08107ea7b237be35d P fea177e632284c95850ac019913d1bc6: Deleting consensus metadata
Aug 09 06:22:20 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:22:20.779681 18507 catalog_manager.cc:4841] TS 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963): tablet 26c5fbb82ff14ea08107ea7b237be35d (table test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]) successfully deleted
I20260809 06:22:20.779783 18504 catalog_manager.cc:4841] TS fea177e632284c95850ac019913d1bc6 (127.3.137.66:40135): tablet 26c5fbb82ff14ea08107ea7b237be35d (table test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]) successfully deleted
I20260809 06:22:20.783694 19054 ts_tablet_manager.cc:1918] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:20.783867 19054 log.cc:1196] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/26c5fbb82ff14ea08107ea7b237be35d
I20260809 06:22:20.784407 19054 ts_tablet_manager.cc:1939] T 26c5fbb82ff14ea08107ea7b237be35d P 7967018d49534e3f8efee5d807dc9b66: Deleting consensus metadata
I20260809 06:22:20.786175 18507 catalog_manager.cc:4841] TS 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059): tablet 26c5fbb82ff14ea08107ea7b237be35d (table test-table [id=f5248f9e0ee04fb0818b0f0ed8eb93cf]) successfully deleted
I20260809 06:22:20.786914 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.753933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.3.137.65:40059)
0809 06:22:20.754133 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.754140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.761580 (+ 7440us) server_negotiation.cc:407] Connection header received
0809 06:22:20.761818 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.761823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.761902 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.762008 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:20.764213 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.765470 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.771893 (+ 6423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.772527 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.772672 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.780196 (+ 7524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:20.780208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:20.780220 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:20.780256 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:20.783651 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.784533 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.784541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.784545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.784703 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.785101 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.785106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.785109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.785478 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:20.785848 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.786443 (+ 595us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.786646 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:22:20.787609 18716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:20.891884 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18587
I20260809 06:22:20.913527 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18727
Aug 09 06:22:20 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:22:20.931529 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:20.914459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.3.137.66:40135)
0809 06:22:20.914625 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:20.914629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:20.914650 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:22:20.914704 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:20.914708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:20.914785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:20.914875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:20.916601 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.917433 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.919427 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:20.919883 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:20.919979 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:20.925305 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:20.925316 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:20.925332 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:20.925371 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:20.928460 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.929146 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.929153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.929156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.929309 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:20.929715 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:20.929720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:20.929722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:20.930087 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:20.930419 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:20.930734 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 06:22:20.931226 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:22:20.932283 18857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:21.050817 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18727
I20260809 06:22:21.081760 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18867
Aug 09 06:22:21 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:22:21.097091 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:21.082522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.3.137.67:33963)
0809 06:22:21.082707 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:21.082715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:21.082804 (+ 89us) server_negotiation.cc:407] Connection header received
0809 06:22:21.083103 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:21.083108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:21.083196 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:21.083322 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:21.084620 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:21.085345 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:21.086636 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:21.087116 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:21.087252 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:21.091687 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:21.091702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:21.091719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:21.091753 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:21.094222 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:21.094995 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:21.095006 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:21.095010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:21.095157 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:21.095586 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:21.095592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:21.095595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:21.095936 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:21.096240 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:21.096537 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 06:22:21.096759 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:22:21.097745 19002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:21.185956 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18867
I20260809 06:22:21.204943 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 18487
I20260809 06:22:21.206173 18555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:21.316385 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 18487
I20260809 06:22:21.340081 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--webserver_interface=127.3.137.126
--webserver_port=36655
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:21.671361 19061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:21.671690 19061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:21.671854 19061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:21.679147 19061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:21.679283 19061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:21.679394 19061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:21.679502 19061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:21.679591 19061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:21.689465 19061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:42343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=36655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:21.690765 19061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:21.692111 19061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:21.716382 19071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:21.716323 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:21.716426 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:21.717073 19061 server_base.cc:1034] running on GCE node
I20260809 06:22:21.717999 19061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:21.719740 19061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:21.720950 19061 hybrid_clock.cc:648] HybridClock initialized: now 1786256541720904 us; error 42 us; skew 500 ppm
I20260809 06:22:21.721379 19061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:21 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:21.725000 19061 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:21.726729 19061 webserver.cc:460] Webserver started at http://127.3.137.126:36655/ using document root <none> and password file <none>
I20260809 06:22:21.727591 19061 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:21.727768 19061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:21.733778 19061 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 06:22:21.737028 19078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:21.738435 19061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:22:21.738642 19061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "9f1d5e9706254fb89e1806365522e518"
format_stamp: "Formatted at 2026-08-09 06:22:18 on dist-test-slave-2w3w"
server_key: "97f6b05a5aeaccee9a90be79f73d4adc"
server_key_iv: "c1514b7741b1e6eb4e6ae099e59c1ee1"
server_key_version: "encryptionkey@0"
I20260809 06:22:21.739852 19061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:21.757051 19061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:21.760905 19061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:21.761185 19061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:21.780253 19061 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:42343
I20260809 06:22:21.780279 19134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:42343 every 8 connection(s)
I20260809 06:22:21.781791 19061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:21.786688 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19061
I20260809 06:22:21.787752 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:40059
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=34431
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:21.789752 19135 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:22:21.795636 19135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap starting.
I20260809 06:22:21.819134 19135 log.cc:824] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:21.832510 19135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:22:21.833112 19135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap complete.
I20260809 06:22:21.843839 19135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:21.844818 19135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Initialized, Role: FOLLOWER
I20260809 06:22:21.845402 19135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:21.845678 19135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:21.845824 19135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:21.846004 19135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:22:21.849961 19135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:21.850481 19135 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1d5e9706254fb89e1806365522e518; no voters:
I20260809 06:22:21.851167 19135 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:22:21.851363 19139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:22:21.854943 19135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:21.856820 19139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 LEADER]: Becoming Leader. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Running, Role: LEADER
I20260809 06:22:21.857601 19139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:21.860693 19141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1d5e9706254fb89e1806365522e518. Latest consensus state: current_term: 2 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:21.861123 19141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:21.860862 19140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:21.861497 19140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:21.862918 19149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:21.872964 19149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:21.874809 19149 catalog_manager.cc:1259] Loaded cluster ID: 1cd631c1964b4ea3b263e86da66ac75d
I20260809 06:22:21.874915 19149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:21.879141 19149 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 06:22:21.903121 19149 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 06:22:21.910472 19149 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:21.911868 19149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Loaded TSK: 0
I20260809 06:22:21.913080 19149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:22:22.161908 19137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:22.162213 19137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:22.162375 19137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:22.169620 19137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:22.169772 19137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:22.169938 19137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:22.179538 19137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=34431
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:22.180758 19137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:22.182034 19137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:22.196472 19163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:22.196583 19167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:22.197084 19165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:22.198007 19137 server_base.cc:1034] running on GCE node
I20260809 06:22:22.198671 19137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:22.199780 19137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:22.201097 19137 hybrid_clock.cc:648] HybridClock initialized: now 1786256542201014 us; error 74 us; skew 500 ppm
I20260809 06:22:22.201436 19137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:22 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:22.205116 19137 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:22.206945 19137 webserver.cc:460] Webserver started at http://127.3.137.65:34431/ using document root <none> and password file <none>
I20260809 06:22:22.207917 19137 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:22.208081 19137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:22.215363 19137 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 06:22:22.218604 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.220299 19137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:22.220460 19137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "7967018d49534e3f8efee5d807dc9b66"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "1116aacacd3b777c78b77d2343f42405"
server_key_iv: "38617ee99b6244d3ad60bbec658a96b6"
server_key_version: "encryptionkey@0"
I20260809 06:22:22.221830 19137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:22.253932 19137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:22.259729 19137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:22.260052 19137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:22.261315 19137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:22.263053 19137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:22.263167 19137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.263309 19137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:22.263396 19137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.313787 19137 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40059
I20260809 06:22:22.313855 19295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40059 every 8 connection(s)
I20260809 06:22:22.315318 19137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:22.319443 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19137
I20260809 06:22:22.320240 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:40135
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=42959
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:22 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:22.345276 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:22.318364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:47723 (local address 127.3.137.126:42343)
0809 06:22:22.323483 (+ 5119us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:22.323499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:22.323549 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:22:22.323890 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:22.323910 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:22.324339 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:22.324726 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:22.326532 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.328566 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.330884 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.331600 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.331741 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:22.336436 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:22.336445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:22.336471 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:22.336514 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:22.340327 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:22.341306 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:22.341315 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:22.341323 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:22.341530 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:22.342078 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:22.342083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:22.342086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:22.342710 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:22.342885 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:22.343852 (+ 967us) server_negotiation.cc:299] Negotiation successful
0809 06:22:22.344285 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4866,"thread_start_us":320,"threads_started":1}
I20260809 06:22:22.345638 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:22.318635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:47723)
0809 06:22:22.319547 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:22.319613 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:22.322307 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:22.324912 (+ 2605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:22.324923 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:22.325596 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:22.326268 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:22.326289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.328750 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.328757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:22.330703 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:22.330722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.331840 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.331848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:22.332129 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:22.332935 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:22.332964 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:22.336137 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:22.340502 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:22.340514 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:22.340531 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:22.341161 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:22.341669 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:22.341676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:22.341682 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:22.341943 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:22.342879 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:22.342890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:22.343326 (+ 436us) client_negotiation.cc:769] Sending connection context
0809 06:22:22.343643 (+ 317us) client_negotiation.cc:240] Negotiation successful
0809 06:22:22.344224 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":344,"threads_started":1}
I20260809 06:22:22.348809 19296 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:22.349150 19296 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:22.350067 19296 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:22.352721 19098 ts_manager.cc:194] Registered new tserver with Master: 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:22.354761 19098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:47723
W20260809 06:22:22.652057 19300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:22.652313 19300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:22.652403 19300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:22.659504 19300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:22.659601 19300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:22.659710 19300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:22.668892 19300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=42959
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:22.670054 19300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:22.671314 19300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:22.684034 19306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:22.684034 19305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:22.684319 19308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:22.684969 19300 server_base.cc:1034] running on GCE node
I20260809 06:22:22.685539 19300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:22.686473 19300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:22.687682 19300 hybrid_clock.cc:648] HybridClock initialized: now 1786256542687616 us; error 51 us; skew 500 ppm
I20260809 06:22:22.688022 19300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:22 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:22.691186 19300 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:22.692561 19300 webserver.cc:460] Webserver started at http://127.3.137.66:42959/ using document root <none> and password file <none>
I20260809 06:22:22.693249 19300 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:22.693410 19300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:22.698904 19300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260809 06:22:22.701897 19316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.703194 19300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:22.703363 19300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "fea177e632284c95850ac019913d1bc6"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "9b2f3fbe9edfbcf9a5bca508698014ff"
server_key_iv: "e4ecc55f13eb5280dfb646a1bf26159e"
server_key_version: "encryptionkey@0"
I20260809 06:22:22.704335 19300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:22.720948 19300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:22.725193 19300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:22.725471 19300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:22.726409 19300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:22.727937 19300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:22.728076 19300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.728211 19300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:22.728307 19300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:22.762337 19300 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40135
I20260809 06:22:22.762391 19434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40135 every 8 connection(s)
I20260809 06:22:22.763885 19300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:22.770809 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19300
I20260809 06:22:22.771544 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:33963
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=40961
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:22:22 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:22.786374 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:22.766708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:34157 (local address 127.3.137.126:42343)
0809 06:22:22.766893 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:22.766898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:22.768162 (+ 1264us) server_negotiation.cc:407] Connection header received
0809 06:22:22.769980 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:22.769985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:22.770071 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:22.770182 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:22.771751 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.773046 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.774574 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.775437 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.775596 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:22.778857 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:22.778866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:22.778874 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:22.778923 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:22.783138 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:22.783907 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:22.783913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:22.783917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:22.784056 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:22.784519 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:22.784526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:22.784530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:22.784845 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:22.785038 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:22.785739 (+ 701us) server_negotiation.cc:299] Negotiation successful
0809 06:22:22.785984 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:22:22.787472 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:22.767274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:34157)
0809 06:22:22.767992 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:22.768032 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:22.769716 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:22.770414 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:22.770421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:22.770916 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:22.771537 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:22.771554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.773221 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.773225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:22.774411 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:22.774423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:22.775644 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:22.775648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:22.775850 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:22.776520 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:22.776541 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:22.778638 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:22.783293 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:22.783303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:22.783318 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:22.783751 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:22.784177 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:22.784184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:22.784188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:22.784392 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:22.784985 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:22.784995 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:22.785337 (+ 342us) client_negotiation.cc:769] Sending connection context
0809 06:22:22.785639 (+ 302us) client_negotiation.cc:240] Negotiation successful
0809 06:22:22.786166 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260809 06:22:22.789124 19435 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:22.789461 19435 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:22.790391 19435 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:22.791960 19098 ts_manager.cc:194] Registered new tserver with Master: fea177e632284c95850ac019913d1bc6 (127.3.137.66:40135)
I20260809 06:22:22.792845 19098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:34157
W20260809 06:22:23.109640 19439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:23.109906 19439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:23.110101 19439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:23.117435 19439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:23.117564 19439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:23.117751 19439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:23.127322 19439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=40961
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:23.128641 19439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:23.129947 19439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:23.143764 19444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:23.143850 19445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:23.143791 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:23.145164 19439 server_base.cc:1034] running on GCE node
I20260809 06:22:23.145678 19439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:23.146476 19439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:23.147665 19439 hybrid_clock.cc:648] HybridClock initialized: now 1786256543147620 us; error 34 us; skew 500 ppm
I20260809 06:22:23.147935 19439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:23 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:23.150936 19439 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:23.152458 19439 webserver.cc:460] Webserver started at http://127.3.137.67:40961/ using document root <none> and password file <none>
I20260809 06:22:23.153138 19439 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:23.153292 19439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:23.158435 19439 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:22:23.161120 19455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:23.162353 19439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 06:22:23.162521 19439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "064718086ad5496085fc62f5697360b4"
format_stamp: "Formatted at 2026-08-09 06:22:20 on dist-test-slave-2w3w"
server_key: "579fffd5aae6cb479157c0c3093aebe6"
server_key_iv: "7394709dcaac20332ba983dc45b64d39"
server_key_version: "encryptionkey@0"
I20260809 06:22:23.163533 19439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:23.181685 19439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:23.185745 19439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:23.186020 19439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:23.186923 19439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:23.188143 19439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:23.188279 19439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:23.188402 19439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:23.188490 19439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:23.222141 19439 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33963
I20260809 06:22:23.222227 19575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33963 every 8 connection(s)
I20260809 06:22:23.223627 19439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:23.230656 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19439
Aug 09 06:22:23 dist-test-slave-2w3w krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:23.245510 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.226010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:57937 (local address 127.3.137.126:42343)
0809 06:22:23.226234 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.226240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.227428 (+ 1188us) server_negotiation.cc:407] Connection header received
0809 06:22:23.229245 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.229249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.229316 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.229404 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:23.230778 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.231783 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.233736 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.234396 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.234543 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.238399 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:23.238406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:23.238412 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:23.238456 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:23.241851 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.243031 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.243036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.243042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.243225 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.243722 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.243728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.243731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.244101 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:23.244258 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.244944 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.245157 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:22:23.246515 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.226490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:57937)
0809 06:22:23.227275 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.227312 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.229009 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.229599 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.229607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.230069 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:23.230600 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.230617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.232025 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.232030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.233584 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.233597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.234632 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.234641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.235002 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.235675 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:23.235705 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:23.238159 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:23.241990 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:23.241996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:23.242006 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:23.242869 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:23.243344 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:23.243350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:23.243357 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:23.243603 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:23.244255 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:23.244264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:23.244587 (+ 323us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.244848 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.245335 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":271,"threads_started":1}
I20260809 06:22:23.246771 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.231648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.3.137.126:42343)
0809 06:22:23.232190 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.232195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.232507 (+ 312us) server_negotiation.cc:407] Connection header received
0809 06:22:23.232637 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.232642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.232715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.232826 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:23.234564 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.235675 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.237179 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.237667 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.237761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.240509 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:23.240517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:23.240521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:23.240572 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:23.243958 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.244699 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.244706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.244709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.244837 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.245237 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.245242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.245246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.245545 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:23.245798 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.246187 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.246397 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":264,"threads_started":1}
I20260809 06:22:23.248046 19576 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42343
I20260809 06:22:23.248402 19576 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:23.249251 19576 heartbeater.cc:511] Master 127.3.137.126:42343 requested a full tablet report, sending...
I20260809 06:22:23.250592 19097 ts_manager.cc:194] Registered new tserver with Master: 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963)
I20260809 06:22:23.251464 19097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:57937
I20260809 06:22:23.261430 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:22:23.268277 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.262324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.3.137.126:42343)
0809 06:22:23.262475 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.262479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.263030 (+ 551us) server_negotiation.cc:407] Connection header received
0809 06:22:23.263230 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.263234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.263352 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.263476 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:22:23.263851 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.264507 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.265998 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.266478 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.266615 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.267209 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:22:23.267683 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:22:23.267841 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.267949 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.268033 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:22:23.269088 19097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:22:23.276175 19097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:22:23.310506 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.287291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:40135 (local address 127.0.0.1:41254)
0809 06:22:23.301895 (+ 14604us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.301921 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.302083 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.303360 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.303373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.303414 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:23.303919 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.303936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.305859 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.305865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.307500 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.307520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.309504 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.309510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.309724 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.309812 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.309929 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.310119 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14448,"thread_start_us":15688,"threads_started":1}
I20260809 06:22:23.311291 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.288035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.3.137.66:40135)
0809 06:22:23.301179 (+ 13144us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.301191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.302246 (+ 1055us) server_negotiation.cc:407] Connection header received
0809 06:22:23.302394 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.302406 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.302776 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.303038 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:23.304136 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.305720 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.307668 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.309215 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.309358 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.309564 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.310799 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.311004 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12923,"thread_start_us":13148,"threads_started":1}
I20260809 06:22:23.312518 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.299574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36376)
0809 06:22:23.302926 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.302949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.303361 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.304858 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.304863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.304913 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:23.305464 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.305476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.307382 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.307388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.309495 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.309507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.311748 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.311754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.311904 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.311942 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.312022 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.312115 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3190,"mutex_wait_us":183,"spinlock_wait_cycles":10368,"thread_start_us":2016,"threads_started":1}
I20260809 06:22:23.312556 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.294941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.3.137.65:40059)
0809 06:22:23.304288 (+ 9347us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.304297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.304325 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:22:23.304429 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.304435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.304644 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.304989 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:23.305646 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.307235 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.309684 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.311470 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.311664 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.311840 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.312083 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.312245 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9140,"thread_start_us":8992,"threads_started":1}
I20260809 06:22:23.314864 19368 tablet_service.cc:1467] Processing CreateTablet for tablet be2a4a7b34e4476da374ea6186de664c (DEFAULT_TABLE table=test-table [id=bc32b6b68bb84e4d92e358e9c6fc7453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:23.315753 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.287000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33963 (local address 127.0.0.1:44918)
0809 06:22:23.294241 (+ 7241us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.294278 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.294430 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.304858 (+ 10428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.304874 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.304910 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:23.305354 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.305364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.307516 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.307523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.309874 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.309889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.311607 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.311633 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.311840 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.311875 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.311942 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.312081 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7009,"mutex_wait_us":69,"spinlock_wait_cycles":1280,"thread_start_us":6850,"threads_started":1}
I20260809 06:22:23.315584 19228 tablet_service.cc:1467] Processing CreateTablet for tablet be2a4a7b34e4476da374ea6186de664c (DEFAULT_TABLE table=test-table [id=bc32b6b68bb84e4d92e358e9c6fc7453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:23.316282 19368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a4a7b34e4476da374ea6186de664c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:23.316944 19228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a4a7b34e4476da374ea6186de664c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:23.318109 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.287263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.3.137.67:33963)
0809 06:22:23.304308 (+ 17045us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.304317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.304345 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:22:23.304428 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.304435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.304643 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.305532 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:23.305836 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.307373 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.310061 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.311338 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.316279 (+ 4941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.316502 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.316567 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.316715 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16531,"thread_start_us":16378,"threads_started":1}
I20260809 06:22:23.320927 19507 tablet_service.cc:1467] Processing CreateTablet for tablet be2a4a7b34e4476da374ea6186de664c (DEFAULT_TABLE table=test-table [id=bc32b6b68bb84e4d92e358e9c6fc7453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:23.322062 19507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a4a7b34e4476da374ea6186de664c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:23.331508 19595 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Bootstrap starting.
I20260809 06:22:23.334499 19595 tablet_bootstrap.cc:654] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:23.335327 19596 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Bootstrap starting.
I20260809 06:22:23.336265 19595 log.cc:824] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:23.338130 19597 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Bootstrap starting.
I20260809 06:22:23.340323 19596 tablet_bootstrap.cc:654] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:23.341061 19595 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: No bootstrap required, opened a new log
I20260809 06:22:23.341333 19595 ts_tablet_manager.cc:1397] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260809 06:22:23.341547 19597 tablet_bootstrap.cc:654] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:23.341593 19596 log.cc:824] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:23.343003 19597 log.cc:824] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:23.344882 19596 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: No bootstrap required, opened a new log
I20260809 06:22:23.345142 19596 ts_tablet_manager.cc:1397] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260809 06:22:23.346081 19597 tablet_bootstrap.cc:492] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: No bootstrap required, opened a new log
I20260809 06:22:23.346338 19597 ts_tablet_manager.cc:1397] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 06:22:23.349463 19595 raft_consensus.cc:348] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.350018 19595 raft_consensus.cc:374] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:23.350178 19595 raft_consensus.cc:729] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7967018d49534e3f8efee5d807dc9b66, State: Initialized, Role: FOLLOWER
I20260809 06:22:23.350814 19595 consensus_queue.cc:260] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.351979 19296 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:23.352579 19595 ts_tablet_manager.cc:1428] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260809 06:22:23.355806 19597 raft_consensus.cc:348] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.356264 19597 raft_consensus.cc:374] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:23.356391 19597 raft_consensus.cc:729] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064718086ad5496085fc62f5697360b4, State: Initialized, Role: FOLLOWER
I20260809 06:22:23.356587 19596 raft_consensus.cc:348] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.357038 19596 raft_consensus.cc:374] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:23.357156 19596 raft_consensus.cc:729] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea177e632284c95850ac019913d1bc6, State: Initialized, Role: FOLLOWER
I20260809 06:22:23.357014 19597 consensus_queue.cc:260] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.357720 19596 consensus_queue.cc:260] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.358150 19576 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:23.358747 19597 ts_tablet_manager.cc:1428] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260809 06:22:23.358879 19435 heartbeater.cc:503] Master 127.3.137.126:42343 was elected leader, sending a full tablet report...
I20260809 06:22:23.359508 19596 ts_tablet_manager.cc:1428] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260809 06:22:23.374459 19603 raft_consensus.cc:482] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:22:23.374661 19603 raft_consensus.cc:504] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.376163 19603 leader_election.cc:288] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963), 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:23.384182 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.376599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:59857 (local address 127.3.137.67:33963)
0809 06:22:23.376826 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.376833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.377193 (+ 360us) server_negotiation.cc:407] Connection header received
0809 06:22:23.377498 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.377503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.377577 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.377645 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:23.378822 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.379903 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.382052 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.383308 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.383420 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.383479 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.383919 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.384007 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 06:22:23.384385 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.376378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:33963 (local address 127.3.137.66:59857)
0809 06:22:23.377089 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.377109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.377311 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.377862 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.377867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.377895 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:23.378544 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.378564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.380030 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.380036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.381890 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.381902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.383525 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.383532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.383752 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.383789 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.383892 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.384021 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":467,"threads_started":1}
I20260809 06:22:23.385475 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.377646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:35775 (local address 127.3.137.65:40059)
0809 06:22:23.377942 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.377949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.379046 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 06:22:23.379206 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.379219 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.379296 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.379401 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:23.380030 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.381117 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.383261 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.384512 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.384661 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.384744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.385100 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.385223 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 06:22:23.385646 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.377242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.3.137.66:35775)
0809 06:22:23.378895 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.378914 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:23.379070 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:23.379425 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:23.379431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:23.379456 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:23.379826 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.379836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.381259 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.381262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.383104 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:23.383120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.384793 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.384802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:23.384941 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.384966 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:22:23.385075 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 06:22:23.385193 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":1262,"threads_started":1}
I20260809 06:22:23.385536 19528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a4a7b34e4476da374ea6186de664c" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064718086ad5496085fc62f5697360b4" is_pre_election: true
I20260809 06:22:23.386051 19528 raft_consensus.cc:2454] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 0.
I20260809 06:22:23.386720 19320 leader_election.cc:302] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064718086ad5496085fc62f5697360b4, fea177e632284c95850ac019913d1bc6; no voters:
I20260809 06:22:23.386663 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a4a7b34e4476da374ea6186de664c" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7967018d49534e3f8efee5d807dc9b66" is_pre_election: true
I20260809 06:22:23.387145 19603 raft_consensus.cc:2781] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:22:23.387137 19250 raft_consensus.cc:2454] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 0.
I20260809 06:22:23.387300 19603 raft_consensus.cc:482] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:22:23.387418 19603 raft_consensus.cc:3037] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:23.389710 19603 raft_consensus.cc:504] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.390295 19603 leader_election.cc:288] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963), 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059)
I20260809 06:22:23.390656 19528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a4a7b34e4476da374ea6186de664c" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064718086ad5496085fc62f5697360b4"
I20260809 06:22:23.390719 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a4a7b34e4476da374ea6186de664c" candidate_uuid: "fea177e632284c95850ac019913d1bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7967018d49534e3f8efee5d807dc9b66"
I20260809 06:22:23.390849 19528 raft_consensus.cc:3037] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:23.390950 19250 raft_consensus.cc:3037] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:23.393200 19528 raft_consensus.cc:2454] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 1.
I20260809 06:22:23.393309 19250 raft_consensus.cc:2454] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea177e632284c95850ac019913d1bc6 in term 1.
I20260809 06:22:23.393631 19320 leader_election.cc:302] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064718086ad5496085fc62f5697360b4, fea177e632284c95850ac019913d1bc6; no voters:
I20260809 06:22:23.393965 19603 raft_consensus.cc:2781] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:23.394408 19603 raft_consensus.cc:686] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 1 LEADER]: Becoming Leader. State: Replica: fea177e632284c95850ac019913d1bc6, State: Running, Role: LEADER
I20260809 06:22:23.394860 19603 consensus_queue.cc:237] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } }
I20260809 06:22:23.399541 19098 catalog_manager.cc:5495] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fea177e632284c95850ac019913d1bc6 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "fea177e632284c95850ac019913d1bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fea177e632284c95850ac019913d1bc6" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20260809 06:22:23.420976 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.415595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.3.137.66:40135)
0809 06:22:23.415863 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.415868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.415889 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:22:23.416002 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.416006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.416078 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.416229 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:22:23.416623 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.417342 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.418761 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.419239 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.419379 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.419864 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:22:23.420358 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:22:23.420515 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.420628 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.420706 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:22:23.428936 19250 raft_consensus.cc:1261] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Refusing update from remote peer fea177e632284c95850ac019913d1bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:23.428964 19528 raft_consensus.cc:1261] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Refusing update from remote peer fea177e632284c95850ac019913d1bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:23.429883 19603 consensus_queue.cc:1035] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "064718086ad5496085fc62f5697360b4" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:23.430198 19608 consensus_queue.cc:1035] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7967018d49534e3f8efee5d807dc9b66" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:23.441210 19610 mvcc.cc:205] Tried to move back new op lower bound from 7316506801875050496 to 7316506801749131264. Current Snapshot: MvccSnapshot[applied={T|T < 7316506801875050496}]
I20260809 06:22:23.455793 19098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:23.456138 19098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:23.460114 19098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Sending DeleteTablet for 3 replicas of tablet be2a4a7b34e4476da374ea6186de664c
I20260809 06:22:23.461285 19368 tablet_service.cc:1514] Processing DeleteTablet for tablet be2a4a7b34e4476da374ea6186de664c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:23 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:41254
I20260809 06:22:23.461580 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19137
I20260809 06:22:23.461601 19228 tablet_service.cc:1514] Processing DeleteTablet for tablet be2a4a7b34e4476da374ea6186de664c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:23 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:36376
I20260809 06:22:23.461632 19507 tablet_service.cc:1514] Processing DeleteTablet for tablet be2a4a7b34e4476da374ea6186de664c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:23 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:44918
I20260809 06:22:23.473851 19622 tablet_replica.cc:330] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: stopping tablet replica
I20260809 06:22:23.474052 19624 tablet_replica.cc:330] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: stopping tablet replica
I20260809 06:22:23.474393 19625 tablet_replica.cc:330] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: stopping tablet replica
I20260809 06:22:23.474761 19624 raft_consensus.cc:2229] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:23.475405 19622 raft_consensus.cc:2229] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:22:23.475425 19624 raft_consensus.cc:2258] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:23.474895 19625 raft_consensus.cc:2229] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:23.476033 19625 raft_consensus.cc:2258] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:23.476732 19622 raft_consensus.cc:2258] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:23.478842 19622 ts_tablet_manager.cc:1905] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:23.480556 19625 ts_tablet_manager.cc:1905] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:23.484908 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.463257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.3.137.65:40059)
0809 06:22:23.463464 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.463472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.463493 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:22:23.463555 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.463562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.463655 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.463766 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:23.466317 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.467907 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.470687 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.471649 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.471799 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.476110 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:23.476124 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:23.476140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:23.476178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:23.480082 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.480787 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.480794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.480796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.480913 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.481304 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.481309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.481313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.481860 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:23.482171 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.484276 (+ 2105us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.484531 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:22:23.485952 19291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:23.486299 19622 ts_tablet_manager.cc:1918] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:23.486577 19622 log.cc:1196] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/be2a4a7b34e4476da374ea6186de664c
I20260809 06:22:23.486795 19624 ts_tablet_manager.cc:1905] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:23.487464 19622 ts_tablet_manager.cc:1939] T be2a4a7b34e4476da374ea6186de664c P fea177e632284c95850ac019913d1bc6: Deleting consensus metadata
I20260809 06:22:23.489760 19079 catalog_manager.cc:4841] TS fea177e632284c95850ac019913d1bc6 (127.3.137.66:40135): tablet be2a4a7b34e4476da374ea6186de664c (table test-table [id=bc32b6b68bb84e4d92e358e9c6fc7453]) successfully deleted
I20260809 06:22:23.494052 19624 ts_tablet_manager.cc:1918] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:23.494284 19624 log.cc:1196] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/be2a4a7b34e4476da374ea6186de664c
I20260809 06:22:23.494921 19624 ts_tablet_manager.cc:1939] T be2a4a7b34e4476da374ea6186de664c P 064718086ad5496085fc62f5697360b4: Deleting consensus metadata
I20260809 06:22:23.496783 19082 catalog_manager.cc:4841] TS 064718086ad5496085fc62f5697360b4 (127.3.137.67:33963): tablet be2a4a7b34e4476da374ea6186de664c (table test-table [id=bc32b6b68bb84e4d92e358e9c6fc7453]) successfully deleted
I20260809 06:22:23.582033 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19137
W20260809 06:22:23.604532 19082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:40059 (error 108)
I20260809 06:22:23.604862 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19300
I20260809 06:22:23.607834 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.607330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36404)
0809 06:22:23.607525 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.607641 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 06:22:23.608263 19082 catalog_manager.cc:4584] TS 7967018d49534e3f8efee5d807dc9b66 (127.3.137.65:40059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be2a4a7b34e4476da374ea6186de664c: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
I20260809 06:22:23.621225 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.605864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.3.137.66:40135)
0809 06:22:23.606121 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.606130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.606163 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:22:23.606323 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.606333 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.606467 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.606586 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:23.608810 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.609743 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.611295 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.611805 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.611966 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.614867 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:23.614885 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:23.614900 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:23.614947 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:23.618088 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.618856 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.618865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.618869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.619201 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.619601 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.619609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.619614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.619959 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:23.620321 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.620631 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.620852 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 06:22:23.621863 19430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:23.662025 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.661564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36420)
0809 06:22:23.661761 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.661855 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 06:22:23.663285 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.662777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36426)
0809 06:22:23.662945 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.663094 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 06:22:23.723790 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.723311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36432)
0809 06:22:23.723541 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.723627 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 06:22:23.724776 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.724434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36448)
0809 06:22:23.724594 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.724670 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 06:22:23.741895 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19300
I20260809 06:22:23.762509 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19439
I20260809 06:22:23.780467 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.763335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.3.137.67:33963)
0809 06:22:23.763528 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:23.763534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:23.763634 (+ 100us) server_negotiation.cc:407] Connection header received
0809 06:22:23.763852 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:23.763863 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:23.763928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:23.764034 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:23.765463 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.766189 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.767761 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:23.768163 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:23.768299 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:23.775013 (+ 6714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:23.775027 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:23.775044 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:23.775081 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:23.777540 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.778238 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.778245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.778247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.778392 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:23.778776 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:23.778781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:23.778784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:23.779281 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:23.779577 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:23.779932 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 06:22:23.780162 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:22:23.781098 19571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:23.805078 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.804639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36458)
0809 06:22:23.804836 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.804917 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:22:23.806176 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:23.805764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36474)
0809 06:22:23.805998 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:23.806067 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 06:22:23.879427 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19439
I20260809 06:22:23.897416 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19061
I20260809 06:22:23.898569 19130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:24.013782 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:24.012969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:40059 (local address 127.0.0.1:36482)
0809 06:22:24.013196 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:24.013293 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:22:24.014261 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19061
I20260809 06:22:24.035466 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--webserver_interface=127.3.137.126
--webserver_port=36655
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:24.371294 19633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:24.371552 19633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:24.371640 19633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:24.378676 19633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:24.378778 19633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:24.378826 19633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:24.378883 19633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:24.378928 19633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:24.389686 19633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:42343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=36655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:24.391152 19633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:24.392434 19633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:24.404471 19638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:24.404371 19641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:24.404546 19639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:24.405524 19633 server_base.cc:1034] running on GCE node
I20260809 06:22:24.406283 19633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:24.407711 19633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:24.408941 19633 hybrid_clock.cc:648] HybridClock initialized: now 1786256544408910 us; error 40 us; skew 500 ppm
I20260809 06:22:24.409276 19633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:24 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:24.412379 19633 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:24.413903 19633 webserver.cc:460] Webserver started at http://127.3.137.126:36655/ using document root <none> and password file <none>
I20260809 06:22:24.414575 19633 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:24.414741 19633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:24.420145 19633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:22:24.422935 19648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:24.424299 19633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:22:24.424510 19633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "9f1d5e9706254fb89e1806365522e518"
format_stamp: "Formatted at 2026-08-09 06:22:18 on dist-test-slave-2w3w"
server_key: "97f6b05a5aeaccee9a90be79f73d4adc"
server_key_iv: "c1514b7741b1e6eb4e6ae099e59c1ee1"
server_key_version: "encryptionkey@0"
I20260809 06:22:24.425493 19633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:24.447420 19633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:24.451480 19633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:24.451746 19633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:24.470947 19633 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:42343
I20260809 06:22:24.471010 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:42343 every 8 connection(s)
I20260809 06:22:24.472568 19633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:24.479187 19703 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:22:24.481243 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19633
I20260809 06:22:24.482138 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:40059
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=34431
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:24.483013 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap starting.
I20260809 06:22:24.502936 19703 log.cc:824] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:24.519256 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:22:24.519816 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518: Bootstrap complete.
I20260809 06:22:24.528956 19703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:24.529930 19703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Initialized, Role: FOLLOWER
I20260809 06:22:24.530536 19703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:24.530778 19703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:24.530907 19703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:24.531134 19703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 2 FOLLOWER]: Advancing to term 3
I20260809 06:22:24.534827 19703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:24.535327 19703 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1d5e9706254fb89e1806365522e518; no voters:
I20260809 06:22:24.536019 19703 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 06:22:24.536235 19708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 06:22:24.537395 19708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [term 3 LEADER]: Becoming Leader. State: Replica: 9f1d5e9706254fb89e1806365522e518, State: Running, Role: LEADER
I20260809 06:22:24.538023 19703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:24.538151 19708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } }
I20260809 06:22:24.542331 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:24.542678 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:24.543375 19710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1d5e9706254fb89e1806365522e518. Latest consensus state: current_term: 3 leader_uuid: "9f1d5e9706254fb89e1806365522e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1d5e9706254fb89e1806365522e518" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42343 } } }
I20260809 06:22:24.543728 19710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1d5e9706254fb89e1806365522e518 [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:24.558882 19720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:24.567581 19720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:24.569139 19720 catalog_manager.cc:1259] Loaded cluster ID: 1cd631c1964b4ea3b263e86da66ac75d
I20260809 06:22:24.569459 19720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 06:22:24.571832 19720 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efee277a9ad google::LogMessage::Fail() at ??:0
@ 0x7efee277d88a google::LogMessage::SendToLog() at ??:0
I20260809 06:22:24.578034 19728 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7efee277a4de google::LogMessage::Flush() at ??:0
@ 0x7efee277ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efef81ad512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efef81aa976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efef8232610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efef8232439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efee53c6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efee53c991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efee53c9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efee53a4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efee867e6db start_thread at ??:0
@ 0x7efee161461f clone at ??:0
W20260809 06:22:24.909026 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:24.909300 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:24.909399 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:24.916615 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:24.916725 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:24.916841 19706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:24.926297 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=34431
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:24.927541 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:24.928821 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:24.942051 19750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:24.942024 19751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:24.942111 19753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:24.943090 19706 server_base.cc:1034] running on GCE node
I20260809 06:22:24.943745 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:24.944717 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:24.945955 19706 hybrid_clock.cc:648] HybridClock initialized: now 1786256544945898 us; error 54 us; skew 500 ppm
I20260809 06:22:24.946267 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:24 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:24.949363 19706 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:24.950870 19706 webserver.cc:460] Webserver started at http://127.3.137.65:34431/ using document root <none> and password file <none>
I20260809 06:22:24.951584 19706 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:24.951730 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:24.957275 19706 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 06:22:24.962581 19760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:24.963919 19706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:22:24.964099 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "7967018d49534e3f8efee5d807dc9b66"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "1116aacacd3b777c78b77d2343f42405"
server_key_iv: "38617ee99b6244d3ad60bbec658a96b6"
server_key_version: "encryptionkey@0"
I20260809 06:22:24.965116 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:24.980496 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:24.984407 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:24.984709 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:24.985684 19706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:24.987490 19767 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260809 06:22:24.992817 19767 ts_tablet_manager.cc:1852] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260809 06:22:24.993855 19767 ts_tablet_manager.cc:1905] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:25.001487 19767 ts_tablet_manager.cc:1918] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:25.001668 19767 log.cc:1196] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/be2a4a7b34e4476da374ea6186de664c
I20260809 06:22:25.002163 19767 ts_tablet_manager.cc:1939] T be2a4a7b34e4476da374ea6186de664c P 7967018d49534e3f8efee5d807dc9b66: Deleting consensus metadata
I20260809 06:22:25.002532 19706 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260809 06:22:25.002691 19706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260809 06:22:25.002835 19706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:25.002928 19706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.038305 19706 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:40059
I20260809 06:22:25.038404 19885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:40059 every 8 connection(s)
I20260809 06:22:25.039710 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:25.042316 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19706
I20260809 06:22:25.043058 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:40135
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=42959
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:25.045225 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.042842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:38599)
0809 06:22:25.043741 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.044031 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":367,"threads_started":1}
W20260809 06:22:25.046274 19886 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:42343 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:42343: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
I20260809 06:22:25.047407 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.046900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:47763)
0809 06:22:25.047156 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.047259 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 06:22:25.048599 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.048136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:52847)
0809 06:22:25.048336 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.048434 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 06:22:25.048887 19886 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:22:25.379745 19889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:25.380012 19889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:25.380172 19889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:25.388720 19889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:25.388857 19889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:25.389040 19889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:25.398656 19889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=42959
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:25.399962 19889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:25.401355 19889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:25.414518 19897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:25.414518 19895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:25.414554 19894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:25.415354 19889 server_base.cc:1034] running on GCE node
I20260809 06:22:25.415964 19889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:25.416977 19889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:25.418176 19889 hybrid_clock.cc:648] HybridClock initialized: now 1786256545418136 us; error 41 us; skew 500 ppm
I20260809 06:22:25.418501 19889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:25 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:25.421687 19889 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:25.423112 19889 webserver.cc:460] Webserver started at http://127.3.137.66:42959/ using document root <none> and password file <none>
I20260809 06:22:25.423749 19889 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:25.423847 19889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:25.428902 19889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:22:25.431537 19906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.432928 19889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:25.433061 19889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "fea177e632284c95850ac019913d1bc6"
format_stamp: "Formatted at 2026-08-09 06:22:19 on dist-test-slave-2w3w"
server_key: "9b2f3fbe9edfbcf9a5bca508698014ff"
server_key_iv: "e4ecc55f13eb5280dfb646a1bf26159e"
server_key_version: "encryptionkey@0"
I20260809 06:22:25.433990 19889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:25.449709 19889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:25.453694 19889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:25.453915 19889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:25.454782 19889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:25.456079 19889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:25.456166 19889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.456243 19889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:25.456307 19889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.496212 19889 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:40135
I20260809 06:22:25.496279 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:40135 every 8 connection(s)
I20260809 06:22:25.497669 19889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:25.502526 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.500567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:47853)
0809 06:22:25.501317 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.501558 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260809 06:22:25.502705 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 19889
W20260809 06:22:25.503271 20032 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:42343 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:42343: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
I20260809 06:22:25.503506 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:33963
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:42343
--webserver_port=40961
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:22:25.504266 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.503821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:49331)
0809 06:22:25.504009 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.504108 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 06:22:25.505252 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.504815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:52501)
0809 06:22:25.504993 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.505098 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 06:22:25.505509 20032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:22:25.849793 20035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:25.850147 20035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:25.850276 20035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:25.857340 20035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:25.857441 20035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:25.857604 20035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:25.867344 20035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=40961
--tserver_master_addrs=127.3.137.126:42343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:25.868577 20035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:25.869840 20035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:25.882639 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:25.882685 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:25.882709 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:25.883669 20035 server_base.cc:1034] running on GCE node
I20260809 06:22:25.884305 20035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:25.885167 20035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:25.886360 20035 hybrid_clock.cc:648] HybridClock initialized: now 1786256545886305 us; error 48 us; skew 500 ppm
I20260809 06:22:25.886616 20035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:25 dist-test-slave-2w3w krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:25.889855 20035 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:25.891377 20035 webserver.cc:460] Webserver started at http://127.3.137.67:40961/ using document root <none> and password file <none>
I20260809 06:22:25.892055 20035 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:25.892186 20035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:25.897446 20035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:22:25.900130 20051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.901489 20035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:22:25.901664 20035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "064718086ad5496085fc62f5697360b4"
format_stamp: "Formatted at 2026-08-09 06:22:20 on dist-test-slave-2w3w"
server_key: "579fffd5aae6cb479157c0c3093aebe6"
server_key_iv: "7394709dcaac20332ba983dc45b64d39"
server_key_version: "encryptionkey@0"
I20260809 06:22:25.902634 20035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:25.935580 20035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:25.940092 20035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:25.940385 20035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:25.941401 20035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:25.942766 20035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:25.942899 20035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.943054 20035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:25.943163 20035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:25.976961 20035 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:33963
I20260809 06:22:25.977067 20173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:33963 every 8 connection(s)
I20260809 06:22:25.978473 20035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:25.982689 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20035
I20260809 06:22:25.983438 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.981476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:55687)
0809 06:22:25.982219 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.982480 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":286,"threads_started":1}
W20260809 06:22:25.984261 20174 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:42343 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:42343: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
I20260809 06:22:25.985139 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.984814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:35393)
0809 06:22:25.984960 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.985039 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 06:22:25.985950 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:25.985639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:50315)
0809 06:22:25.985768 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:25.985841 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 06:22:25.986202 20174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:22:26.050181 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:26.049489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:50293)
0809 06:22:26.049922 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:26.050022 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260809 06:22:26.506882 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:26.506044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:33513)
0809 06:22:26.506551 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:26.506654 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":262,"threads_started":1}
I20260809 06:22:26.988118 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:26.986918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:38341)
0809 06:22:26.987591 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:26.987767 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":354,"threads_started":1}
I20260809 06:22:27.052155 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:27.051270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:39147)
0809 06:22:27.051821 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:27.051939 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
I20260809 06:22:27.508487 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:27.507763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:42523)
0809 06:22:27.508233 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:27.508317 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":219,"threads_started":1}
I20260809 06:22:27.989698 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:27.988975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:44905)
0809 06:22:27.989452 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:27.989536 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":242,"threads_started":1}
I20260809 06:22:28.053732 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:28.053074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:50261)
0809 06:22:28.053489 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:28.053571 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260809 06:22:28.509964 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:28.509292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:49001)
0809 06:22:28.509730 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:28.509806 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260809 06:22:28.991344 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:28.990559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:44677)
0809 06:22:28.991102 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:28.991184 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":278,"threads_started":1}
I20260809 06:22:29.055351 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:29.054577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:46835)
0809 06:22:29.055093 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:29.055175 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260809 06:22:29.511659 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:29.510875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:60795)
0809 06:22:29.511382 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:29.511469 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260809 06:22:29.993247 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:29.992427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:40065)
0809 06:22:29.992970 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:29.993062 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260809 06:22:30.057156 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:30.056401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:35567)
0809 06:22:30.056876 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:30.056968 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
I20260809 06:22:30.513393 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:30.512631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:35479)
0809 06:22:30.513103 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:30.513183 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260809 06:22:30.995043 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:30.994239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:39489)
0809 06:22:30.994725 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:30.994808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260809 06:22:31.058660 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:31.057995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:54871)
0809 06:22:31.058424 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:31.058508 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":246,"threads_started":1}
I20260809 06:22:31.515264 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:31.514374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:45833)
0809 06:22:31.514907 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:31.515075 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":269,"threads_started":1}
I20260809 06:22:31.996595 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:31.995884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:54487)
0809 06:22:31.996330 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:31.996420 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260809 06:22:32.060316 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:32.059537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:55029)
0809 06:22:32.060025 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:32.060111 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":221,"threads_started":1}
I20260809 06:22:32.517081 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:32.516215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:43527)
0809 06:22:32.516720 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:32.516824 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":241,"threads_started":1}
I20260809 06:22:32.998329 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:32.997511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:60351)
0809 06:22:32.998035 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:32.998124 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":249,"threads_started":1}
I20260809 06:22:33.061781 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:33.061120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:59627)
0809 06:22:33.061548 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:33.061626 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
I20260809 06:22:33.518833 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:33.518113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:58015)
0809 06:22:33.518566 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:33.518644 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260809 06:22:34.000051 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:33.999328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:51283)
0809 06:22:33.999764 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:33.999892 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260809 06:22:34.063383 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:34.062644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:52003)
0809 06:22:34.063128 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:34.063221 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":220,"threads_started":1}
I20260809 06:22:34.520547 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:34.519814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:33955)
0809 06:22:34.520297 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:34.520375 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260809 06:22:35.001817 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:35.000971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:45987)
0809 06:22:35.001530 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:35.001646 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":300,"threads_started":1}
I20260809 06:22:35.065001 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:35.064293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:53797)
0809 06:22:35.064728 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:35.064824 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260809 06:22:35.522284 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:35.521499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:55879)
0809 06:22:35.522044 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:35.522122 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260809 06:22:36.003628 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:36.002799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:50179)
0809 06:22:36.003378 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:36.003466 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":368,"threads_started":1}
I20260809 06:22:36.066677 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:36.065929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:33777)
0809 06:22:36.066394 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:36.066489 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":262,"threads_started":1}
I20260809 06:22:36.523973 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:36.523263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:46693)
0809 06:22:36.523701 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:36.523800 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260809 06:22:37.005332 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:37.004606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:59907)
0809 06:22:37.005051 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:37.005140 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260809 06:22:37.068460 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:37.067614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:41335)
0809 06:22:37.068101 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:37.068218 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260809 06:22:37.525684 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:37.524983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:47015)
0809 06:22:37.525434 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:37.525535 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260809 06:22:38.007079 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:38.006254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:33437)
0809 06:22:38.006717 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:38.006870 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260809 06:22:38.070086 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:38.069372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:47369)
0809 06:22:38.069804 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:38.069893 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260809 06:22:38.527449 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:38.526663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:45925)
0809 06:22:38.527181 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:38.527272 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":225,"threads_started":1}
I20260809 06:22:39.008671 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:39.007921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:40049)
0809 06:22:39.008382 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:39.008486 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
I20260809 06:22:39.071738 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:39.070896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:56791)
0809 06:22:39.071471 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:39.071551 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":330,"threads_started":1}
I20260809 06:22:39.528936 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:39.528263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:58605)
0809 06:22:39.528675 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:39.528761 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260809 06:22:40.010341 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:40.009639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:34655)
0809 06:22:40.010079 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:40.010167 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260809 06:22:40.073561 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:40.072798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.65:58635)
0809 06:22:40.073319 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:40.073398 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":241,"threads_started":1}
I20260809 06:22:40.530694 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:40.530014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.66:51753)
0809 06:22:40.530421 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:40.530518 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":207,"threads_started":1}
I20260809 06:22:40.993017 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19706
I20260809 06:22:41.012207 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:41.011429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42343 (local address 127.3.137.67:41765)
0809 06:22:41.011930 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:41.012031 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:42343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
I20260809 06:22:41.015836 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:40.994513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.3.137.65:40059)
0809 06:22:40.995134 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:40.995148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:40.995185 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:22:40.996814 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:40.996825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:40.997238 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:40.997540 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:40.999012 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.000297 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.001560 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.002212 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.002349 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:41.009195 (+ 6846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:41.009204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:41.009221 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:41.009268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:41.012657 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.013323 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.013330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.013337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.013511 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.013857 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.013861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.013863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.014430 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:41.014665 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:41.015132 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 06:22:41.015497 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":325,"threads_started":1}
I20260809 06:22:41.017249 19881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:41.106103 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19706
I20260809 06:22:41.126096 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 19889
I20260809 06:22:41.143537 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:41.127158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.3.137.66:40135)
0809 06:22:41.127667 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:41.127681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:41.127717 (+ 36us) server_negotiation.cc:407] Connection header received
0809 06:22:41.129314 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:41.129324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:41.129736 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:41.130031 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:41.131510 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.132742 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.133926 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.134572 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.134702 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:41.137407 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:41.137418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:41.137434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:41.137482 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:41.140554 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.141213 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.141219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.141225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.141389 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.141727 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.141730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.141732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.142220 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:41.142457 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:41.142822 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 06:22:41.143253 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260809 06:22:41.144874 20027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:41.227728 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 19889
I20260809 06:22:41.246443 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20035
I20260809 06:22:41.264478 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:41.247520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.3.137.67:33963)
0809 06:22:41.248150 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:41.248165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:41.248204 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:22:41.249800 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:41.249809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:41.250218 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:41.250567 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:41.252243 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.253484 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.254755 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:41.255477 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:41.255671 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:41.258303 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:41.258309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:41.258322 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:41.258367 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:41.261080 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.261787 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.261799 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.261806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.261972 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:41.262360 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:41.262366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:41.262369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:41.262858 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:41.263167 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:41.263773 (+ 606us) server_negotiation.cc:299] Negotiation successful
0809 06:22:41.264175 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":309,"threads_started":1}
I20260809 06:22:41.265813 20168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:41.349140 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20035
2026-08-09T06:22:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24486 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:22:41 dist-test-slave-2w3w krb5kdc[20235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:22:41 dist-test-slave-2w3w krb5kdc[20235](info): set up 2 sockets
Aug 09 06:22:41 dist-test-slave-2w3w krb5kdc[20235](info): commencing operation
krb5kdc: starting...
W20260809 06:22:42.506139 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:22:42 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:22:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:22:42.774523 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:43.103955 20248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:43.104210 20248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:43.104378 20248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:43.111362 20248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:22:43.111444 20248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:22:43.111529 20248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:22:43.111626 20248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:22:43.111714 20248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:43.111814 20248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:43.111893 20248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:22:43.111972 20248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:22:43.121416 20248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:43.122629 20248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:43.124014 20248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:43.135546 20253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:43.135514 20254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:43.135701 20256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:43.136411 20248 server_base.cc:1034] running on GCE node
I20260809 06:22:43.137187 20248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:43.138473 20248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:43.139717 20248 hybrid_clock.cc:648] HybridClock initialized: now 1786256563139661 us; error 51 us; skew 500 ppm
I20260809 06:22:43.140089 20248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:43 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:43.143349 20248 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:22:43.144635 20248 webserver.cc:460] Webserver started at http://127.3.137.126:40123/ using document root <none> and password file <none>
I20260809 06:22:43.145257 20248 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:43.145342 20248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:43.145586 20248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:43.148474 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "7659cbb5179f4131b19441e9012086dc"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "78796d4216d050dca987915291015c0d"
server_key_iv: "f83a410dcfe7591e68cda11badf0eefb"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.149283 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "7659cbb5179f4131b19441e9012086dc"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "78796d4216d050dca987915291015c0d"
server_key_iv: "f83a410dcfe7591e68cda11badf0eefb"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.154193 20248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:22:43.157509 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:43.158753 20248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:22:43.158895 20248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "7659cbb5179f4131b19441e9012086dc"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "78796d4216d050dca987915291015c0d"
server_key_iv: "f83a410dcfe7591e68cda11badf0eefb"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.159143 20248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:43.187449 20248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:43.191944 20248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:43.192166 20248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:43.210752 20248 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45835
I20260809 06:22:43.210777 20321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45835 every 8 connection(s)
I20260809 06:22:43.212116 20248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:22:43.215727 20322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:43.220593 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20248
I20260809 06:22:43.220846 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:22:43.221702 3621 external_mini_cluster.cc:1428] Setting key 525347683cfa7af683adbb78bb2b7627
I20260809 06:22:43.227191 20322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap starting.
I20260809 06:22:43.230794 20322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:43.232053 20322 log.cc:824] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:43.235461 20322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: No bootstrap required, opened a new log
Aug 09 06:22:43 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:43.243669 20322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:22:43.244119 20322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:43.244244 20322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Initialized, Role: FOLLOWER
I20260809 06:22:43.244997 20322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:22:43.245267 20322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:22:43.245424 20322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:22:43.245630 20322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:43.246189 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:43.223786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.3.137.126:45835)
0809 06:22:43.224589 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:43.224600 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:43.224642 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:22:43.226278 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:43.226299 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:43.226643 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:43.226940 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:43.229003 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.230267 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.232359 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.233467 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.233603 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:43.238477 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:43.238487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:43.238500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:43.238543 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:43.241794 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:43.242554 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:43.242560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:43.242566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:43.242736 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:43.243219 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:43.243223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:43.243225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:43.243728 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:43.244079 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:43.244633 (+ 554us) server_negotiation.cc:299] Negotiation successful
0809 06:22:43.245015 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":379,"threads_started":1}
I20260809 06:22:43.248204 20322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:22:43.248742 20322 leader_election.cc:302] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7659cbb5179f4131b19441e9012086dc; no voters:
I20260809 06:22:43.249322 20322 leader_election.cc:288] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:22:43.249454 20327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:43.250164 20327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 LEADER]: Becoming Leader. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Running, Role: LEADER
I20260809 06:22:43.250720 20327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:22:43.251477 20322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:22:43.257215 20329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7659cbb5179f4131b19441e9012086dc. Latest consensus state: current_term: 1 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:22:43.258379 20329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:43.257982 20328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:22:43.258977 20328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:22:43.259397 20337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:22:43.264583 20337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:22:43.272917 20337 catalog_manager.cc:1347] Generated new cluster ID: 6a1a10df1ed743588bcdf44877558a3d
I20260809 06:22:43.273074 20337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:22:43.312326 20337 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:22:43.313920 20337 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:22:43.327272 20337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Generated new TSK 0
I20260809 06:22:43.328087 20337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:22:43.408638 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:43.746397 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:43.746650 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:43.746749 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:43.755127 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:43.755229 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:43.755345 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:22:43.765236 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:43.766427 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:43.767742 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:43.780138 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:43.780072 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:43.780546 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:43.781002 20349 server_base.cc:1034] running on GCE node
I20260809 06:22:43.781641 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:43.782660 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:43.783866 20349 hybrid_clock.cc:648] HybridClock initialized: now 1786256563783814 us; error 50 us; skew 500 ppm
I20260809 06:22:43.784171 20349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:43 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:43.787222 20349 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:22:43.788630 20349 webserver.cc:460] Webserver started at http://127.3.137.65:37973/ using document root <none> and password file <none>
I20260809 06:22:43.789307 20349 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:43.789458 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:43.789764 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:43.792666 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "2fd51ec4549123531c907aff4b4cee4f"
server_key_iv: "327b922be5696bdf17e101409e1e9bbe"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.793401 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "2fd51ec4549123531c907aff4b4cee4f"
server_key_iv: "327b922be5696bdf17e101409e1e9bbe"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.798422 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:22:43.801991 20364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:43.803293 20349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:22:43.803470 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "2fd51ec4549123531c907aff4b4cee4f"
server_key_iv: "327b922be5696bdf17e101409e1e9bbe"
server_key_version: "encryptionkey@0"
I20260809 06:22:43.803663 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:43.819730 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:43.823524 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:43.823818 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:43.824779 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:43.826103 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:43.826208 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:43.826356 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:43.826429 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:43.860062 20349 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:45869
I20260809 06:22:43.860083 20486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:45869 every 8 connection(s)
I20260809 06:22:43.861347 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:22:43.865659 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20349
I20260809 06:22:43.865903 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:22:43.866793 3621 external_mini_cluster.cc:1428] Setting key 05ff34ee7ebb097936ba50d56166c465
Aug 09 06:22:43 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:43.883745 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:43.864050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:48897 (local address 127.3.137.126:45835)
0809 06:22:43.864468 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:43.864473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:43.865616 (+ 1143us) server_negotiation.cc:407] Connection header received
0809 06:22:43.868028 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:43.868034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:43.868113 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:43.868213 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:43.870146 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.870899 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.873376 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.873762 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.873855 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:43.878155 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:43.878160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:43.878163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:43.878194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:43.880776 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:43.881491 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:43.881498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:43.881506 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:43.881620 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:43.882033 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:43.882038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:43.882042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:43.882291 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:43.882445 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:43.883119 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 06:22:43.883358 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":207,"threads_started":1}
I20260809 06:22:43.884636 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:43.864618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:48897)
0809 06:22:43.865421 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:43.865480 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:43.867778 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:43.868481 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:43.868492 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:43.869167 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:43.869946 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:43.869990 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.871083 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.871092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:43.873222 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:43.873239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:43.874008 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:43.874017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:43.874340 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:43.875125 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:43.875159 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:43.877939 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:43.880918 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:43.880926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:43.880942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:43.881356 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:43.881731 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:43.881739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:43.881742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:43.881920 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:43.882419 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:43.882426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:43.882742 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 06:22:43.883018 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 06:22:43.883533 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":269,"threads_started":1}
I20260809 06:22:43.886211 20488 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:22:43.886505 20488 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:43.887266 20488 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:22:43.889825 20283 ts_manager.cc:194] Registered new tserver with Master: f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869)
I20260809 06:22:43.892038 20283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:48897
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:22:43.929979 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:44.252264 20494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:44.252552 20494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:44.252700 20494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:44.259902 20494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:44.259997 20494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:44.260103 20494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:22:44.269654 20494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:44.270743 20494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:44.272058 20494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:44.284561 20500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:44.284686 20499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:44.285112 20502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:44.286688 20494 server_base.cc:1034] running on GCE node
I20260809 06:22:44.287348 20494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:44.288223 20494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:44.289410 20494 hybrid_clock.cc:648] HybridClock initialized: now 1786256564289359 us; error 54 us; skew 500 ppm
I20260809 06:22:44.289659 20494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:44 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:44.292762 20494 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:22:44.294199 20494 webserver.cc:460] Webserver started at http://127.3.137.66:44649/ using document root <none> and password file <none>
I20260809 06:22:44.294910 20494 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:44.295075 20494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:44.295385 20494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:44.298086 20494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "b115279c79ef4ba5a46c1a103b77333f"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "3166d32a8b850bef16f2f7dd87b1f09d"
server_key_iv: "acc639a804cf0362cb64c7a8accf0f49"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.298817 20494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "b115279c79ef4ba5a46c1a103b77333f"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "3166d32a8b850bef16f2f7dd87b1f09d"
server_key_iv: "acc639a804cf0362cb64c7a8accf0f49"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.303725 20494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:22:44.306864 20509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.308173 20494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:22:44.308399 20494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "b115279c79ef4ba5a46c1a103b77333f"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "3166d32a8b850bef16f2f7dd87b1f09d"
server_key_iv: "acc639a804cf0362cb64c7a8accf0f49"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.308558 20494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:44.323091 20494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:44.326840 20494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:44.327160 20494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:44.328047 20494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:44.329351 20494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:44.329435 20494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.329574 20494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:44.329628 20494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.363674 20494 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41941
I20260809 06:22:44.363713 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41941 every 8 connection(s)
I20260809 06:22:44.365021 20494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:22:44.366191 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20494
I20260809 06:22:44.366415 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:22:44.367070 3621 external_mini_cluster.cc:1428] Setting key 1b4cf900a1af21c53cd8ddf7ad9bdab7
Aug 09 06:22:44 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:44.386551 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.367996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:42157 (local address 127.3.137.126:45835)
0809 06:22:44.368233 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.368258 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.369548 (+ 1290us) server_negotiation.cc:407] Connection header received
0809 06:22:44.371367 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.371375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.371477 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.371586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:44.373061 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.374188 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.375935 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.376702 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.376853 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.380502 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:44.380511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:44.380519 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:44.380565 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:44.383319 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.384218 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.384222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.384224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.384340 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.384816 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.384819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.384821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.385063 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:44.385168 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.386122 (+ 954us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.386287 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 06:22:44.388011 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.368506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:42157)
0809 06:22:44.369350 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:44.369403 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:44.371114 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:44.371805 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:44.371812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:44.372332 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:44.372888 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.372905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.374366 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.374370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.375788 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.375801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.376909 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.376914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.377104 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.377851 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:44.377873 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:44.380260 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:44.383449 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:44.383460 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:44.383478 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:44.384098 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:44.384450 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:44.384457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:44.384463 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:44.384708 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:44.385190 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:44.385200 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:44.385716 (+ 516us) client_negotiation.cc:769] Sending connection context
0809 06:22:44.386022 (+ 306us) client_negotiation.cc:240] Negotiation successful
0809 06:22:44.386544 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":354,"threads_started":1}
I20260809 06:22:44.389461 20633 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:22:44.389760 20633 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:44.390542 20633 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:22:44.392043 20283 ts_manager.cc:194] Registered new tserver with Master: b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941)
I20260809 06:22:44.392839 20283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:42157
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:22:44.431314 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:22:44.758590 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:22:44.758900 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:22:44.759092 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:22:44.766076 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:22:44.766242 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:22:44.766417 20638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:22:44.775579 20638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:22:44.776743 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:22:44.777976 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:22:44.790432 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:44.790536 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:22:44.791004 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:22:44.791491 20638 server_base.cc:1034] running on GCE node
I20260809 06:22:44.792097 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:22:44.792949 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:22:44.794183 20638 hybrid_clock.cc:648] HybridClock initialized: now 1786256564794137 us; error 43 us; skew 500 ppm
I20260809 06:22:44.794517 20638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:22:44 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:22:44.797742 20638 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:22:44.799063 20638 webserver.cc:460] Webserver started at http://127.3.137.67:44619/ using document root <none> and password file <none>
I20260809 06:22:44.799654 20638 fs_manager.cc:359] Metadata directory not provided
I20260809 06:22:44.799749 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:22:44.799995 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:22:44.802911 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "047716f62295473687519bd142778f92"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "895e73d9c8c9792d3884b75d3634257f"
server_key_iv: "94f489c3114078b8283a4d86b32b049b"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.803831 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "047716f62295473687519bd142778f92"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "895e73d9c8c9792d3884b75d3634257f"
server_key_iv: "94f489c3114078b8283a4d86b32b049b"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.808857 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:22:44.812218 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.813488 20638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:22:44.813654 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "047716f62295473687519bd142778f92"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "895e73d9c8c9792d3884b75d3634257f"
server_key_iv: "94f489c3114078b8283a4d86b32b049b"
server_key_version: "encryptionkey@0"
I20260809 06:22:44.813871 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:22:44.825124 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:22:44.829598 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:22:44.829900 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:22:44.831102 20638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:22:44.832561 20638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:22:44.832674 20638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.832835 20638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:22:44.832919 20638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:22:44.870041 20638 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:43811
I20260809 06:22:44.870087 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:43811 every 8 connection(s)
I20260809 06:22:44.871378 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:22:44.877830 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20638
I20260809 06:22:44.878032 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:22:44.878561 3621 external_mini_cluster.cc:1428] Setting key a37459f3e2e3530712ae9d771c1e0f55
Aug 09 06:22:44 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:22:44.892058 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.873936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:46451 (local address 127.3.137.126:45835)
0809 06:22:44.874122 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.874126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.875245 (+ 1119us) server_negotiation.cc:407] Connection header received
0809 06:22:44.877045 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.877051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.877144 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.877257 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:44.878838 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.879933 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.882262 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.882717 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.882822 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.886507 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:44.886511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:44.886515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:44.886547 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:44.889283 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.889950 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.889954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.889957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.890087 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.890460 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.890463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.890465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.890794 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:44.890903 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.891637 (+ 734us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.891806 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:22:44.893409 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.874337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:46451)
0809 06:22:44.875097 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:44.875139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:44.876782 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:44.877511 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:44.877519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:44.878074 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:22:44.878639 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.878682 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.880402 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.880407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.882060 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.882091 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.882928 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.882933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.883212 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.884065 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:22:44.884088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:22:44.886298 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:22:44.889414 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:44.889420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:44.889431 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:44.889844 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:44.890187 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:22:44.890190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:22:44.890192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:22:44.890365 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:22:44.890903 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:22:44.890908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:22:44.891295 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 06:22:44.891535 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 06:22:44.892016 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":290,"threads_started":1}
I20260809 06:22:44.894796 20780 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:22:44.895012 20488 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:22:44.895166 20780 heartbeater.cc:464] Registering TS with master...
I20260809 06:22:44.895977 20780 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:22:44.897461 20283 ts_manager.cc:194] Registered new tserver with Master: 047716f62295473687519bd142778f92 (127.3.137.67:43811)
I20260809 06:22:44.898236 20283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:46451
I20260809 06:22:44.906203 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:22:44.923863 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.910809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.3.137.126:45835)
0809 06:22:44.911048 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.911054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.911409 (+ 355us) server_negotiation.cc:407] Connection header received
0809 06:22:44.911608 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.911614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.911682 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.911785 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:44.913189 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.913930 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.915413 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.915828 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.915967 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.918567 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:44.918572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:44.918576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:44.918605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:44.921133 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.921778 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.921786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.921792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.921938 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:44.922326 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:44.922332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:44.922335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:44.922673 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:44.922941 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.923316 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.923532 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:22:44.927616 20283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:22:44.934602 20283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:22:44.958031 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.946605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41941 (local address 127.0.0.1:42518)
0809 06:22:44.948345 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:44.948362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:44.948483 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:44.949457 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:44.949464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:44.949508 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:44.949995 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.950021 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.952371 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.952380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.955345 (+ 2965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.955361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.956869 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.956876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.957072 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.957129 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 06:22:44.957301 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 06:22:44.957481 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"mutex_wait_us":131,"thread_start_us":1429,"threads_started":1}
I20260809 06:22:44.958110 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.947087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.3.137.66:41941)
0809 06:22:44.948815 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.948824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.948855 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:22:44.948951 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.948958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.949215 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.950248 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:44.950473 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.952206 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.955545 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.956665 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.956821 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.957021 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.957556 (+ 535us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.957734 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":1977,"threads_started":1}
I20260809 06:22:44.958626 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.946816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.3.137.65:45869)
0809 06:22:44.949063 (+ 2247us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.949071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.949121 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:22:44.949217 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.949224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.949454 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.950386 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:44.950636 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.952332 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.955570 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.957083 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.957216 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.957418 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.958044 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.958271 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":2349,"threads_started":1}
I20260809 06:22:44.958649 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.946501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45869 (local address 127.0.0.1:34914)
0809 06:22:44.947447 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:44.947482 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:44.947653 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:44.949662 (+ 2009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:44.949673 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:44.949692 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:44.950178 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.950189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.952472 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.952478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.955349 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.955361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.957309 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.957316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.957487 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.957517 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:22:44.957604 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:22:44.957731 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":157,"thread_start_us":557,"threads_started":1}
I20260809 06:22:44.962572 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.948749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43811 (local address 127.0.0.1:46886)
0809 06:22:44.953709 (+ 4960us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:44.953727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:44.953840 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:44.955311 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:44.955322 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:44.955351 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:44.955838 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.955850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.957729 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.957734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.960516 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:44.960531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.961746 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.961753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:44.961941 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.961978 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:22:44.962079 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 06:22:44.962246 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4798,"thread_start_us":4654,"threads_started":1}
I20260809 06:22:44.962694 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:44.949469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.3.137.67:43811)
0809 06:22:44.954404 (+ 4935us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:44.954413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:44.954444 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:22:44.954527 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:44.954534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:44.954757 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:44.954941 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:44.956047 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.957564 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.960669 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:44.961541 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:44.961694 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:44.961927 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:44.962137 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 06:22:44.962352 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4761,"thread_start_us":4610,"threads_started":1}
I20260809 06:22:44.962196 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 (DEFAULT_TABLE table=test-table [id=5992506f84494dfbbf6ec8663af5a608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:44.963768 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5caf033736c74d3ab9992c99ee2f1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:44.963284 20415 tablet_service.cc:1467] Processing CreateTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 (DEFAULT_TABLE table=test-table [id=5992506f84494dfbbf6ec8663af5a608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:44.964668 20415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5caf033736c74d3ab9992c99ee2f1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:44.966138 20708 tablet_service.cc:1467] Processing CreateTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 (DEFAULT_TABLE table=test-table [id=5992506f84494dfbbf6ec8663af5a608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:22:44.967265 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5caf033736c74d3ab9992c99ee2f1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:22:44.979282 20800 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Bootstrap starting.
I20260809 06:22:44.982129 20802 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Bootstrap starting.
I20260809 06:22:44.982489 20800 tablet_bootstrap.cc:654] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:44.983665 20801 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Bootstrap starting.
I20260809 06:22:44.983840 20800 log.cc:824] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:44.985853 20802 tablet_bootstrap.cc:654] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:44.987125 20802 log.cc:824] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:44.987388 20801 tablet_bootstrap.cc:654] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Neither blocks nor log segments found. Creating new log.
I20260809 06:22:44.988580 20800 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: No bootstrap required, opened a new log
I20260809 06:22:44.988660 20801 log.cc:824] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:22:44.988873 20800 ts_tablet_manager.cc:1397] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260809 06:22:44.990209 20802 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: No bootstrap required, opened a new log
I20260809 06:22:44.990451 20802 ts_tablet_manager.cc:1397] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:22:44.991602 20801 tablet_bootstrap.cc:492] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: No bootstrap required, opened a new log
I20260809 06:22:44.991850 20801 ts_tablet_manager.cc:1397] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:22:44.997686 20802 raft_consensus.cc:348] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:44.998214 20802 raft_consensus.cc:374] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:44.998349 20802 raft_consensus.cc:729] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b115279c79ef4ba5a46c1a103b77333f, State: Initialized, Role: FOLLOWER
I20260809 06:22:44.999003 20802 consensus_queue.cc:260] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.000159 20633 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:22:44.999915 20800 raft_consensus.cc:348] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.000341 20800 raft_consensus.cc:374] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:45.000429 20800 raft_consensus.cc:729] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047716f62295473687519bd142778f92, State: Initialized, Role: FOLLOWER
I20260809 06:22:45.001011 20802 ts_tablet_manager.cc:1428] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260809 06:22:45.001034 20800 consensus_queue.cc:260] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.002844 20801 raft_consensus.cc:348] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.003389 20801 raft_consensus.cc:374] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:22:45.003486 20801 raft_consensus.cc:729] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21b10e970bd4e4dbcde6eb37311a31c, State: Initialized, Role: FOLLOWER
I20260809 06:22:45.003751 20780 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:22:45.004081 20801 consensus_queue.cc:260] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.006260 20800 ts_tablet_manager.cc:1428] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260809 06:22:45.007400 20801 ts_tablet_manager.cc:1428] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260809 06:22:45.096993 20809 raft_consensus.cc:482] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:22:45.097287 20809 raft_consensus.cc:504] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.099144 20809 leader_election.cc:288] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941), 047716f62295473687519bd142778f92 (127.3.137.67:43811)
I20260809 06:22:45.106364 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.099482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:52123 (local address 127.3.137.66:41941)
0809 06:22:45.099697 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.099705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.100827 (+ 1122us) server_negotiation.cc:407] Connection header received
0809 06:22:45.101037 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.101041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.101098 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.101197 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:45.101901 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.102617 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.104597 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.105361 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.105538 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.105638 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.105941 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.106079 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:22:45.106390 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.099995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:46595 (local address 127.3.137.67:43811)
0809 06:22:45.100191 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.100197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.100966 (+ 769us) server_negotiation.cc:407] Connection header received
0809 06:22:45.101174 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.101179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.101234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.101333 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:22:45.102025 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.102752 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.104614 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.105376 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.105589 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.105670 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.105988 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.106105 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:22:45.106518 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.099612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43811 (local address 127.3.137.65:46595)
0809 06:22:45.100859 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:45.100877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:45.101036 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:45.101350 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:45.101354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:45.101390 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:45.101845 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:45.101855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.102916 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.102921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:45.104450 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:45.104463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.105669 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.105676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:45.105821 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.105843 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:22:45.105946 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 06:22:45.106068 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":1,"thread_start_us":507,"threads_started":1}
I20260809 06:22:45.106899 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.099322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41941 (local address 127.3.137.65:52123)
0809 06:22:45.100692 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0809 06:22:45.100720 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:22:45.100872 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:22:45.101219 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:22:45.101223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:22:45.101250 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:22:45.101644 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:45.101679 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.102752 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.102759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:45.104428 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:22:45.104440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.105617 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.105624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:22:45.105783 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.105807 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:22:45.105913 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:22:45.106038 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":51,"thread_start_us":1171,"threads_started":1}
I20260809 06:22:45.107577 20730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5caf033736c74d3ab9992c99ee2f1333" candidate_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047716f62295473687519bd142778f92" is_pre_election: true
I20260809 06:22:45.107877 20586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5caf033736c74d3ab9992c99ee2f1333" candidate_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b115279c79ef4ba5a46c1a103b77333f" is_pre_election: true
I20260809 06:22:45.108109 20730 raft_consensus.cc:2454] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21b10e970bd4e4dbcde6eb37311a31c in term 0.
I20260809 06:22:45.108417 20586 raft_consensus.cc:2454] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21b10e970bd4e4dbcde6eb37311a31c in term 0.
I20260809 06:22:45.108875 20366 leader_election.cc:302] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047716f62295473687519bd142778f92, f21b10e970bd4e4dbcde6eb37311a31c; no voters:
I20260809 06:22:45.109331 20809 raft_consensus.cc:2781] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:22:45.109485 20809 raft_consensus.cc:482] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:22:45.109586 20809 raft_consensus.cc:3037] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:45.111879 20809 raft_consensus.cc:504] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.112618 20809 leader_election.cc:288] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [CANDIDATE]: Term 1 election: Requested vote from peers b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941), 047716f62295473687519bd142778f92 (127.3.137.67:43811)
I20260809 06:22:45.112946 20586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5caf033736c74d3ab9992c99ee2f1333" candidate_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b115279c79ef4ba5a46c1a103b77333f"
I20260809 06:22:45.113117 20730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5caf033736c74d3ab9992c99ee2f1333" candidate_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047716f62295473687519bd142778f92"
I20260809 06:22:45.113163 20586 raft_consensus.cc:3037] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:22:45.113363 20730 raft_consensus.cc:3037] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Advancing to term 1
W20260809 06:22:45.113761 20489 tablet.cc:2369] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:45.115628 20586 raft_consensus.cc:2454] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21b10e970bd4e4dbcde6eb37311a31c in term 1.
I20260809 06:22:45.116077 20367 leader_election.cc:302] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b115279c79ef4ba5a46c1a103b77333f, f21b10e970bd4e4dbcde6eb37311a31c; no voters:
I20260809 06:22:45.116115 20730 raft_consensus.cc:2454] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21b10e970bd4e4dbcde6eb37311a31c in term 1.
I20260809 06:22:45.116405 20809 raft_consensus.cc:2781] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:22:45.116922 20809 raft_consensus.cc:686] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 1 LEADER]: Becoming Leader. State: Replica: f21b10e970bd4e4dbcde6eb37311a31c, State: Running, Role: LEADER
W20260809 06:22:45.116995 20634 tablet.cc:2369] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:45.117549 20809 consensus_queue.cc:237] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } }
I20260809 06:22:45.122212 20283 catalog_manager.cc:5495] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c reported cstate change: term changed from 0 to 1, leader changed from <none> to f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65). New cstate: current_term: 1 leader_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } health_report { overall_health: UNKNOWN } } }
W20260809 06:22:45.122867 20781 tablet.cc:2369] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:22:45.149603 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.143914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.3.137.65:45869)
0809 06:22:45.144113 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.144121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.144211 (+ 90us) server_negotiation.cc:407] Connection header received
0809 06:22:45.144479 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.144483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.144575 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.144725 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:22:45.145113 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.145821 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.147476 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.147872 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.148045 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.148520 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:22:45.149022 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:22:45.149174 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.149268 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.149345 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:22:45.157212 20586 raft_consensus.cc:1261] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Refusing update from remote peer f21b10e970bd4e4dbcde6eb37311a31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:45.157226 20730 raft_consensus.cc:1261] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Refusing update from remote peer f21b10e970bd4e4dbcde6eb37311a31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:22:45.158177 20812 consensus_queue.cc:1035] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:45.158545 20809 consensus_queue.cc:1035] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:22:45.172166 20815 mvcc.cc:205] Tried to move back new op lower bound from 7316506890874945536 to 7316506890725199872. Current Snapshot: MvccSnapshot[applied={T|T < 7316506890874945536}]
I20260809 06:22:45.186223 20282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:45.186509 20282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:22:45.190148 20282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Sending DeleteTablet for 3 replicas of tablet 5caf033736c74d3ab9992c99ee2f1333
I20260809 06:22:45.191409 20415 tablet_service.cc:1514] Processing DeleteTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:45 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:34914
I20260809 06:22:45.191540 20708 tablet_service.cc:1514] Processing DeleteTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:45 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:46886
I20260809 06:22:45.191645 20561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5caf033736c74d3ab9992c99ee2f1333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:22:45 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:42518
I20260809 06:22:45.192978 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20349
I20260809 06:22:45.206856 20824 tablet_replica.cc:330] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: stopping tablet replica
I20260809 06:22:45.207484 20824 raft_consensus.cc:2229] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:22:45.208304 20824 raft_consensus.cc:2258] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:45.211059 20824 ts_tablet_manager.cc:1905] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:45.219596 20824 ts_tablet_manager.cc:1918] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:45.219780 20824 log.cc:1196] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/5caf033736c74d3ab9992c99ee2f1333
I20260809 06:22:45.220350 20824 ts_tablet_manager.cc:1939] T 5caf033736c74d3ab9992c99ee2f1333 P f21b10e970bd4e4dbcde6eb37311a31c: Deleting consensus metadata
I20260809 06:22:45.221490 20826 tablet_replica.cc:330] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: stopping tablet replica
I20260809 06:22:45.221974 20826 raft_consensus.cc:2229] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:45.222362 20264 catalog_manager.cc:4841] TS f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869): tablet 5caf033736c74d3ab9992c99ee2f1333 (table test-table [id=5992506f84494dfbbf6ec8663af5a608]) successfully deleted
I20260809 06:22:45.222537 20826 raft_consensus.cc:2258] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:45.228206 20826 ts_tablet_manager.cc:1905] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:22:45 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:22:45.235826 20825 tablet_replica.cc:330] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: stopping tablet replica
I20260809 06:22:45.236334 20825 raft_consensus.cc:2229] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:22:45.236917 20825 raft_consensus.cc:2258] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:22:45.240108 20826 ts_tablet_manager.cc:1918] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:45.240296 20826 log.cc:1196] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/5caf033736c74d3ab9992c99ee2f1333
I20260809 06:22:45.241072 20826 ts_tablet_manager.cc:1939] T 5caf033736c74d3ab9992c99ee2f1333 P b115279c79ef4ba5a46c1a103b77333f: Deleting consensus metadata
I20260809 06:22:45.241793 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.194511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.3.137.65:45869)
0809 06:22:45.194752 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.194761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.218079 (+ 23318us) server_negotiation.cc:407] Connection header received
0809 06:22:45.218385 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.218392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.218499 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.218571 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:45.223010 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.223914 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.225303 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.226112 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.226282 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.232760 (+ 6478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:45.232778 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:45.232798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:45.232848 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:45.237539 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.238466 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.238475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.238478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.238662 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.239173 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.239180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.239186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.239739 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:45.240061 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.241147 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.241414 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:22:45.242096 20825 ts_tablet_manager.cc:1905] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:22:45.242540 20482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:45.242997 20267 catalog_manager.cc:4841] TS b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941): tablet 5caf033736c74d3ab9992c99ee2f1333 (table test-table [id=5992506f84494dfbbf6ec8663af5a608]) successfully deleted
I20260809 06:22:45.249336 20825 ts_tablet_manager.cc:1918] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:22:45.249505 20825 log.cc:1196] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/5caf033736c74d3ab9992c99ee2f1333
I20260809 06:22:45.249948 20825 ts_tablet_manager.cc:1939] T 5caf033736c74d3ab9992c99ee2f1333 P 047716f62295473687519bd142778f92: Deleting consensus metadata
I20260809 06:22:45.251569 20265 catalog_manager.cc:4841] TS 047716f62295473687519bd142778f92 (127.3.137.67:43811): tablet 5caf033736c74d3ab9992c99ee2f1333 (table test-table [id=5992506f84494dfbbf6ec8663af5a608]) successfully deleted
I20260809 06:22:45.343690 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20349
I20260809 06:22:45.363085 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20494
Aug 09 06:22:45 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:22:45.379362 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.364018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.3.137.66:41941)
0809 06:22:45.364209 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.364215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.364234 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:22:45.364334 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.364340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.364425 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.364535 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:45.365894 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.366677 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.368326 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.368776 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.368927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.373395 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:45.373411 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:45.373423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:45.373455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:45.376453 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.377139 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.377148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.377155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.377286 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.377688 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.377694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.377697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.378117 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:45.378409 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.378741 (+ 332us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.379027 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:22:45.380053 20628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:45.470176 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20494
I20260809 06:22:45.489483 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20638
Aug 09 06:22:45 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:22:45.507648 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:22:45.490294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.3.137.67:43811)
0809 06:22:45.490469 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:22:45.490473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:22:45.490530 (+ 57us) server_negotiation.cc:407] Connection header received
0809 06:22:45.490688 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:22:45.490692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:22:45.490773 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:22:45.490861 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:22:45.492279 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.493020 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.494316 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:22:45.494722 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:22:45.494817 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:22:45.499724 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:22:45.499744 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:22:45.499775 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:22:45.499830 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:22:45.503791 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.504714 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.504725 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.504728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.504900 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:22:45.505473 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:22:45.505481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:22:45.505486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:22:45.505955 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:22:45.506308 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0809 06:22:45.506998 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 06:22:45.507258 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:22:45.508365 20775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:45.596125 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20638
I20260809 06:22:45.615201 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20248
I20260809 06:22:45.616375 20317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:22:45.727900 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20248
I20260809 06:23:05.754215 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--webserver_interface=127.3.137.126
--webserver_port=40123
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:06.107175 20832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:06.107479 20832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:06.107643 20832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:06.115080 20832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:23:06.115180 20832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:23:06.115218 20832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:23:06.115264 20832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:06.115310 20832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:06.115377 20832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:06.115423 20832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:06.115469 20832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:06.125392 20832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=40123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:06.126614 20832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:06.127879 20832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:06.152486 20843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:06.152443 20840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:06.152534 20841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:06.156607 20832 server_base.cc:1034] running on GCE node
I20260809 06:23:06.157538 20832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:06.159269 20832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:06.160533 20832 hybrid_clock.cc:648] HybridClock initialized: now 1786256586160463 us; error 40 us; skew 500 ppm
I20260809 06:23:06.160936 20832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:06 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:06.164731 20832 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:23:06.166536 20832 webserver.cc:460] Webserver started at http://127.3.137.126:40123/ using document root <none> and password file <none>
I20260809 06:23:06.167428 20832 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:06.167586 20832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:06.173573 20832 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 06:23:06.176591 20850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:06.178066 20832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:23:06.178297 20832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "7659cbb5179f4131b19441e9012086dc"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "78796d4216d050dca987915291015c0d"
server_key_iv: "f83a410dcfe7591e68cda11badf0eefb"
server_key_version: "encryptionkey@0"
I20260809 06:23:06.179577 20832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:06.196601 20832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:06.201164 20832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:06.201508 20832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:06.222680 20832 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45835
I20260809 06:23:06.222719 20909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45835 every 8 connection(s)
I20260809 06:23:06.224316 20832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:06.231319 20910 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:23:06.234575 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20832
I20260809 06:23:06.234851 20910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap starting.
I20260809 06:23:06.235653 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:45869
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=37973
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:23:06.258520 20910 log.cc:824] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:06.271164 20910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:23:06.271740 20910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap complete.
I20260809 06:23:06.281119 20910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:06.281935 20910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Initialized, Role: FOLLOWER
I20260809 06:23:06.282549 20910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:06.282797 20910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:06.282924 20910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:06.283160 20910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:23:06.287351 20910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:06.287873 20910 leader_election.cc:302] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7659cbb5179f4131b19441e9012086dc; no voters:
I20260809 06:23:06.288555 20910 leader_election.cc:288] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:23:06.288718 20914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:23:06.291388 20910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:06.295900 20914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 LEADER]: Becoming Leader. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Running, Role: LEADER
I20260809 06:23:06.296581 20914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:06.299579 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:23:06.300020 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:06.300575 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7659cbb5179f4131b19441e9012086dc. Latest consensus state: current_term: 2 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:23:06.300941 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:06.312256 20924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:06.320587 20924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:06.322091 20924 catalog_manager.cc:1259] Loaded cluster ID: 6a1a10df1ed743588bcdf44877558a3d
I20260809 06:23:06.322269 20924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:06.326385 20924 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:06.327982 20924 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Loaded TSK: 0
I20260809 06:23:06.364532 20924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Generated new TSK 1
I20260809 06:23:06.365001 20924 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:23:06.598148 20912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:06.598403 20912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:06.598542 20912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:06.605615 20912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:06.605715 20912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:06.605902 20912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:06.615656 20912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=37973
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:06.616834 20912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:06.618096 20912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:06.632429 20938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:06.632491 20939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:06.633078 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:06.633522 20912 server_base.cc:1034] running on GCE node
I20260809 06:23:06.634159 20912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:06.635188 20912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:06.636415 20912 hybrid_clock.cc:648] HybridClock initialized: now 1786256586636368 us; error 44 us; skew 500 ppm
I20260809 06:23:06.636837 20912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:06 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:06.640059 20912 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:23:06.641629 20912 webserver.cc:460] Webserver started at http://127.3.137.65:37973/ using document root <none> and password file <none>
I20260809 06:23:06.642287 20912 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:06.642437 20912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:06.647951 20912 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:23:06.650674 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:06.652108 20912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:23:06.652271 20912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "2fd51ec4549123531c907aff4b4cee4f"
server_key_iv: "327b922be5696bdf17e101409e1e9bbe"
server_key_version: "encryptionkey@0"
I20260809 06:23:06.653239 20912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:06.679263 20912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:06.683537 20912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:06.683862 20912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:06.685055 20912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:06.686396 20912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:06.686542 20912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:06.686676 20912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:06.686774 20912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:06.721383 20912 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:45869
I20260809 06:23:06.721482 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:45869 every 8 connection(s)
I20260809 06:23:06.722906 20912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:06.728430 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 20912
I20260809 06:23:06.729199 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:41941
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=44649
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:23:06 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:06.748785 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:06.725835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:46351 (local address 127.3.137.126:45835)
0809 06:23:06.726362 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:06.726372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:06.727120 (+ 748us) server_negotiation.cc:407] Connection header received
0809 06:23:06.729906 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:06.729928 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:06.730227 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:06.730494 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:06.732485 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:06.733801 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:06.736351 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:06.736967 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:06.737098 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:06.741302 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:06.741309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:06.741322 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:06.741359 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:06.744272 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:06.745059 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:06.745066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:06.745071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:06.745236 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:06.745687 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:06.745691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:06.745693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:06.746145 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:06.746389 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:06.747160 (+ 771us) server_negotiation.cc:299] Negotiation successful
0809 06:23:06.747535 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260809 06:23:06.748782 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:06.726070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:46351)
0809 06:23:06.726937 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:06.727005 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:06.729354 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:06.730707 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:06.730721 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:06.731543 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:06.732291 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:06.732318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:06.733982 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:06.733990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:06.736204 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:06.736222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:06.737203 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:06.737213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:06.737523 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:06.738263 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:06.738297 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:06.741009 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:06.744470 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:06.744479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:06.744493 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:06.744907 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:06.745353 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:06.745359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:06.745362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:06.745552 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:06.746293 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:06.746301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:06.746636 (+ 335us) client_negotiation.cc:769] Sending connection context
0809 06:23:06.746899 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 06:23:06.747477 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":375,"threads_started":1}
I20260809 06:23:06.751873 21070 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:23:06.752163 21070 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:06.752826 21070 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:23:06.755326 20869 ts_manager.cc:194] Registered new tserver with Master: f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869)
I20260809 06:23:06.757714 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:46351
W20260809 06:23:07.070744 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:07.071092 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:07.071271 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:07.078336 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:07.078433 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:07.078626 21074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:07.088466 21074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:41941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=44649
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:07.089711 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:07.091075 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:07.104512 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:07.104606 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:07.105060 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:07.105523 21074 server_base.cc:1034] running on GCE node
I20260809 06:23:07.106105 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:07.107070 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:07.108297 21074 hybrid_clock.cc:648] HybridClock initialized: now 1786256587108244 us; error 43 us; skew 500 ppm
I20260809 06:23:07.108605 21074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:07 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:07.116357 21074 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:23:07.117868 21074 webserver.cc:460] Webserver started at http://127.3.137.66:44649/ using document root <none> and password file <none>
I20260809 06:23:07.118587 21074 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:07.118752 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:07.123998 21074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:23:07.126596 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.127863 21074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:23:07.128059 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "b115279c79ef4ba5a46c1a103b77333f"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "3166d32a8b850bef16f2f7dd87b1f09d"
server_key_iv: "acc639a804cf0362cb64c7a8accf0f49"
server_key_version: "encryptionkey@0"
I20260809 06:23:07.129004 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:07.157590 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:07.161597 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:07.161872 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:07.162818 21074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:07.164281 21074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:07.164413 21074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.164541 21074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:07.164628 21074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.199308 21074 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41941
I20260809 06:23:07.199414 21208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41941 every 8 connection(s)
I20260809 06:23:07.200785 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:07.210949 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21074
I20260809 06:23:07.211670 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:43811
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=44619
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:23:07 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:07.220943 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.203737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:38035 (local address 127.3.137.126:45835)
0809 06:23:07.203920 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.203925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.205147 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 06:23:07.206870 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.206875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.206947 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.207100 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:07.208708 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.209497 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.211033 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.211550 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.211685 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.215437 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:07.215442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:07.215445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:07.215497 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:07.217995 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.218699 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.218706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.218709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.218836 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.219284 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.219290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.219293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.219568 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:07.219729 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.220369 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.220613 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:23:07.222021 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.204266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:38035)
0809 06:23:07.204977 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.205024 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.206610 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.207373 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.207386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.207906 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:07.208548 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.208566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.209633 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.209636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.210827 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.210839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.211780 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.211788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.212064 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.212756 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:07.212777 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:07.215220 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:07.218145 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:07.218153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:07.218167 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:07.218579 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:07.218949 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:07.218982 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:07.218985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:07.219166 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:07.219707 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:07.219715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:07.220020 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.220270 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.220735 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":279,"threads_started":1}
I20260809 06:23:07.224107 21209 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:23:07.224442 21209 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:07.225124 21209 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:23:07.226719 20869 ts_manager.cc:194] Registered new tserver with Master: b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941)
I20260809 06:23:07.227836 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:38035
W20260809 06:23:07.546042 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:07.546343 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:07.546528 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:07.553963 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:07.554131 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:07.554335 21212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:07.564220 21212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=44619
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:07.565460 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:07.566756 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:07.579656 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:07.579697 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:07.579607 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:07.580837 21212 server_base.cc:1034] running on GCE node
I20260809 06:23:07.581408 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:07.582271 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:07.583534 21212 hybrid_clock.cc:648] HybridClock initialized: now 1786256587583483 us; error 40 us; skew 500 ppm
I20260809 06:23:07.583868 21212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:07 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:07.587123 21212 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:23:07.588536 21212 webserver.cc:460] Webserver started at http://127.3.137.67:44619/ using document root <none> and password file <none>
I20260809 06:23:07.589171 21212 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:07.589296 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:07.594528 21212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:23:07.597254 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.598642 21212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:23:07.598860 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "047716f62295473687519bd142778f92"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "895e73d9c8c9792d3884b75d3634257f"
server_key_iv: "94f489c3114078b8283a4d86b32b049b"
server_key_version: "encryptionkey@0"
I20260809 06:23:07.600015 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:07.617929 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:07.622182 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:07.622452 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:07.623474 21212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:07.624866 21212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:07.624962 21212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.625113 21212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:07.625213 21212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:07.659919 21212 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:43811
I20260809 06:23:07.660036 21351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:43811 every 8 connection(s)
I20260809 06:23:07.661345 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:07.664041 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21212
Aug 09 06:23:07 dist-test-slave-2w3w krb5kdc[20235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:07.681564 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.665037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.3.137.126:45835)
0809 06:23:07.666426 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.666430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.666445 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:23:07.666496 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.666500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.666556 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.666644 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:07.668965 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.669830 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.671445 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.671962 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.672068 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.675147 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:07.675152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:07.675155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:07.675185 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:07.678352 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.679289 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.679293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.679296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.679419 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.679896 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.679901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.679903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.680314 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:07.680626 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.680889 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.681161 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":1193,"threads_started":1}
I20260809 06:23:07.683754 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.663834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:35645 (local address 127.3.137.126:45835)
0809 06:23:07.664013 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.664018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.665529 (+ 1511us) server_negotiation.cc:407] Connection header received
0809 06:23:07.668366 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.668370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.668434 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.668525 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:07.670091 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.670876 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.672845 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.673239 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.673366 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.676760 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:07.676765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:07.676768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:07.676807 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:07.680043 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.681050 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.681055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.681059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.681220 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.681667 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.681673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.681679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.682014 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:07.682158 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.683124 (+ 966us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.683403 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:23:07.684872 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.664427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:35645)
0809 06:23:07.665352 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.665406 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.668032 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.668799 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.668810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.669353 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:07.669915 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.669940 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.671060 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.671065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.672707 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.672720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.673480 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.673487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.673696 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.674319 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:07.674340 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:07.676555 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:07.680200 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:07.680209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:07.680251 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:07.680899 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:07.681341 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:07.681346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:07.681350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:07.681541 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:07.682177 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:07.682185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:07.682677 (+ 492us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.683010 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.683525 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":413,"threads_started":1}
I20260809 06:23:07.686300 21352 heartbeater.cc:347] Connected to a master server at 127.3.137.126:45835
I20260809 06:23:07.686568 21352 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:07.687278 21352 heartbeater.cc:511] Master 127.3.137.126:45835 requested a full tablet report, sending...
I20260809 06:23:07.688606 20869 ts_manager.cc:194] Registered new tserver with Master: 047716f62295473687519bd142778f92 (127.3.137.67:43811)
I20260809 06:23:07.689502 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:35645
I20260809 06:23:07.696722 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:23:07.703598 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.697573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.3.137.126:45835)
0809 06:23:07.697717 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.697721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.698338 (+ 617us) server_negotiation.cc:407] Connection header received
0809 06:23:07.698526 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.698529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.698655 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.698768 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:07.699261 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.700091 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.701713 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.702175 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.702267 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.702789 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:07.703095 (+ 306us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 06:23:07.703410 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59888: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 06:23:07.703784 21072 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59888: token expired
W20260809 06:23:07.704468 20787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.137.126:45835)
I20260809 06:23:07.718060 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.705023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.3.137.126:45835)
0809 06:23:07.705169 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.705173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.705278 (+ 105us) server_negotiation.cc:407] Connection header received
0809 06:23:07.705394 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.705397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.705478 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.705566 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:07.707213 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.707985 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.709653 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.710050 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.710139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.713247 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:07.713251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:07.713254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:07.713282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:07.715906 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.716535 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.716539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.716541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.716653 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.716994 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.716997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.716999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.717273 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:07.717459 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.717691 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.717848 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:23:07.721417 20869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:23:07.729367 20869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:23:07.751847 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.741234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.3.137.66:41941)
0809 06:23:07.742373 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.742381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.743249 (+ 868us) server_negotiation.cc:407] Connection header received
0809 06:23:07.743906 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.743915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.744142 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.744343 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.745125 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.746743 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.749123 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.750555 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.750717 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.750926 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.751317 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.751517 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":843,"threads_started":1}
I20260809 06:23:07.751935 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.740674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41941 (local address 127.0.0.1:59902)
0809 06:23:07.742944 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.742984 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.743121 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.744351 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.744356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.744381 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.744896 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.744909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.746941 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.746949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.748955 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.748970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.750820 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.750827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.751044 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.751109 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.751269 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.751460 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"mutex_wait_us":94,"thread_start_us":1899,"threads_started":1}
I20260809 06:23:07.752596 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.740790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.3.137.67:43811)
0809 06:23:07.742138 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.742147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.742180 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:23:07.742291 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.742302 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.743513 (+ 1211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.743717 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.744787 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.746144 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.749395 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.750743 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.750859 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.751084 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.752108 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.752302 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":954,"threads_started":1}
I20260809 06:23:07.753445 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.740624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43811 (local address 127.0.0.1:55326)
0809 06:23:07.741513 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.741567 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.741739 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.744026 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.744032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.744063 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.744519 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.744535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.746348 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.746353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.749212 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.749233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.751153 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.751161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.751365 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.751400 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.751493 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.751625 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":141,"thread_start_us":400,"threads_started":1}
I20260809 06:23:07.753482 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.742136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45869 (local address 127.0.0.1:40490)
0809 06:23:07.743320 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.743343 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.743499 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.744680 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.744685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.744716 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.745191 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.745205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.747119 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.747124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.749472 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.749482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.751862 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.751869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.752796 (+ 927us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.752839 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.752940 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.753069 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":44,"thread_start_us":820,"threads_started":1}
I20260809 06:23:07.753837 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.742783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.3.137.65:45869)
0809 06:23:07.744149 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.744154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.744174 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:23:07.744242 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.744246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.744477 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.744674 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.745382 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.746899 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.750078 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.751578 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.752379 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.752612 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.752988 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.753164 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":1000,"threads_started":1}
I20260809 06:23:07.755815 21141 tablet_service.cc:1467] Processing CreateTablet for tablet bb8d32f9aedd4881b53d67096239e95d (DEFAULT_TABLE table=test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:07.757366 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8d32f9aedd4881b53d67096239e95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:07.758291 21001 tablet_service.cc:1467] Processing CreateTablet for tablet bb8d32f9aedd4881b53d67096239e95d (DEFAULT_TABLE table=test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:07.758616 21281 tablet_service.cc:1467] Processing CreateTablet for tablet bb8d32f9aedd4881b53d67096239e95d (DEFAULT_TABLE table=test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:07.759747 21001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8d32f9aedd4881b53d67096239e95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:07.759934 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8d32f9aedd4881b53d67096239e95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:07.761785 21070 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:23:07.772464 21369 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Bootstrap starting.
I20260809 06:23:07.775663 21370 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Bootstrap starting.
I20260809 06:23:07.776356 21369 tablet_bootstrap.cc:654] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:07.777221 21372 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Bootstrap starting.
I20260809 06:23:07.777982 21369 log.cc:824] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:07.779798 21370 tablet_bootstrap.cc:654] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:07.780380 21372 tablet_bootstrap.cc:654] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:07.781091 21370 log.cc:824] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:07.781256 21369 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: No bootstrap required, opened a new log
I20260809 06:23:07.781476 21372 log.cc:824] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:07.781517 21369 ts_tablet_manager.cc:1397] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:23:07.784162 21370 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: No bootstrap required, opened a new log
I20260809 06:23:07.784439 21370 ts_tablet_manager.cc:1397] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:23:07.784641 21372 tablet_bootstrap.cc:492] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: No bootstrap required, opened a new log
I20260809 06:23:07.784888 21372 ts_tablet_manager.cc:1397] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 06:23:07.789863 21369 raft_consensus.cc:348] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.790210 21369 raft_consensus.cc:374] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:07.790341 21369 raft_consensus.cc:729] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047716f62295473687519bd142778f92, State: Initialized, Role: FOLLOWER
I20260809 06:23:07.791039 21369 consensus_queue.cc:260] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.792150 21352 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:23:07.792809 21369 ts_tablet_manager.cc:1428] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260809 06:23:07.795363 21370 raft_consensus.cc:348] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.795420 21372 raft_consensus.cc:348] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.795814 21372 raft_consensus.cc:374] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:07.795888 21370 raft_consensus.cc:374] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:07.795940 21372 raft_consensus.cc:729] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21b10e970bd4e4dbcde6eb37311a31c, State: Initialized, Role: FOLLOWER
I20260809 06:23:07.795992 21370 raft_consensus.cc:729] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b115279c79ef4ba5a46c1a103b77333f, State: Initialized, Role: FOLLOWER
I20260809 06:23:07.796625 21372 consensus_queue.cc:260] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.796625 21370 consensus_queue.cc:260] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.798702 21209 heartbeater.cc:503] Master 127.3.137.126:45835 was elected leader, sending a full tablet report...
I20260809 06:23:07.799494 21370 ts_tablet_manager.cc:1428] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260809 06:23:07.800324 21372 ts_tablet_manager.cc:1428] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260809 06:23:07.862607 21376 raft_consensus.cc:482] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:07.862946 21376 raft_consensus.cc:504] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.864682 21376 leader_election.cc:288] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941), f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869)
I20260809 06:23:07.871618 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.865259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:49685 (local address 127.3.137.66:41941)
0809 06:23:07.865474 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.865481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.865504 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:23:07.865590 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.865598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.865693 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.865812 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.866332 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.867475 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.869616 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.870558 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.870727 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.870806 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.871187 (+ 381us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.871332 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:23:07.871747 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.865392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:47715 (local address 127.3.137.65:45869)
0809 06:23:07.865648 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.865657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.866226 (+ 569us) server_negotiation.cc:407] Connection header received
0809 06:23:07.866401 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.866405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.866469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.866539 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.867220 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.868184 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.870044 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.870802 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.871006 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.871082 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.871375 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.871512 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:23:07.871856 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.864978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41941 (local address 127.3.137.67:49685)
0809 06:23:07.865297 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.865320 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.865456 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.865861 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.865866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.865901 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.866129 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.866138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.867609 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.867618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.869443 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.869456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.870738 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.870744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.870943 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.871037 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.871159 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.871302 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260809 06:23:07.871850 21377 raft_consensus.cc:482] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:07.872411 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.865087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45869 (local address 127.3.137.67:47715)
0809 06:23:07.866109 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.866124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.866264 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.866595 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.866601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.866630 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.867060 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.867073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.868325 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.868332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.869876 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.869888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.871050 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.871057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.871217 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.871240 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.871348 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.871471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":3,"thread_start_us":651,"threads_started":1}
I20260809 06:23:07.872516 21377 raft_consensus.cc:504] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.873307 21161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "047716f62295473687519bd142778f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b115279c79ef4ba5a46c1a103b77333f" is_pre_election: true
I20260809 06:23:07.873440 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "047716f62295473687519bd142778f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" is_pre_election: true
I20260809 06:23:07.873816 21161 raft_consensus.cc:2454] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047716f62295473687519bd142778f92 in term 0.
I20260809 06:23:07.873970 21022 raft_consensus.cc:2454] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047716f62295473687519bd142778f92 in term 0.
I20260809 06:23:07.874532 21377 leader_election.cc:288] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047716f62295473687519bd142778f92 (127.3.137.67:43811), f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869)
I20260809 06:23:07.874531 21231 leader_election.cc:302] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047716f62295473687519bd142778f92, b115279c79ef4ba5a46c1a103b77333f; no voters:
I20260809 06:23:07.875042 21376 raft_consensus.cc:2781] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:23:07.875713 21376 raft_consensus.cc:482] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:23:07.876121 21376 raft_consensus.cc:3037] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:07.879305 21376 raft_consensus.cc:504] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.879964 21376 leader_election.cc:288] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [CANDIDATE]: Term 1 election: Requested vote from peers b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941), f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869)
I20260809 06:23:07.880353 21161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "047716f62295473687519bd142778f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b115279c79ef4ba5a46c1a103b77333f"
I20260809 06:23:07.880538 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "047716f62295473687519bd142778f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
I20260809 06:23:07.880580 21161 raft_consensus.cc:3037] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:07.880746 21022 raft_consensus.cc:3037] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:07.884148 21161 raft_consensus.cc:2454] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047716f62295473687519bd142778f92 in term 1.
I20260809 06:23:07.884217 21022 raft_consensus.cc:2454] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047716f62295473687519bd142778f92 in term 1.
I20260809 06:23:07.884699 21231 leader_election.cc:302] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047716f62295473687519bd142778f92, b115279c79ef4ba5a46c1a103b77333f; no voters:
I20260809 06:23:07.885097 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.875594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:51601 (local address 127.3.137.67:43811)
0809 06:23:07.876046 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.876054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.876465 (+ 411us) server_negotiation.cc:407] Connection header received
0809 06:23:07.876760 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.876766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.876832 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.876912 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.877947 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.878708 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.882246 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.883675 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.883834 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.883905 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.884686 (+ 781us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.884828 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 06:23:07.885267 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.875663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:38503 (local address 127.3.137.65:45869)
0809 06:23:07.875849 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.875855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.876622 (+ 767us) server_negotiation.cc:407] Connection header received
0809 06:23:07.876753 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.876758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.876831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.876927 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:07.877764 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.878906 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.881962 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.883553 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.883806 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.883890 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.884819 (+ 929us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.884970 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:23:07.885289 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.874815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:43811 (local address 127.3.137.66:51601)
0809 06:23:07.876363 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.876383 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.876580 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.877156 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.877160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.877188 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.877803 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.877813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.878849 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.878856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.882048 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.882064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.884226 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.884232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.884461 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.884534 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.884639 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.884850 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":102,"thread_start_us":558,"threads_started":1}
I20260809 06:23:07.885653 21376 raft_consensus.cc:2781] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:07.886188 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.875375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45869 (local address 127.3.137.66:38503)
0809 06:23:07.876504 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:07.876521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:07.876632 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:07.876968 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:07.876973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:07.877003 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:07.877539 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.877557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.879074 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.879079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.881724 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:07.881742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.884159 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.884171 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:07.884456 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.884536 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 06:23:07.884639 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 06:23:07.885766 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":1052,"thread_start_us":562,"threads_started":1}
I20260809 06:23:07.886197 21376 raft_consensus.cc:686] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 1 LEADER]: Becoming Leader. State: Replica: 047716f62295473687519bd142778f92, State: Running, Role: LEADER
I20260809 06:23:07.886281 21304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "b115279c79ef4ba5a46c1a103b77333f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047716f62295473687519bd142778f92" is_pre_election: true
I20260809 06:23:07.886838 21376 consensus_queue.cc:237] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } }
I20260809 06:23:07.888911 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8d32f9aedd4881b53d67096239e95d" candidate_uuid: "b115279c79ef4ba5a46c1a103b77333f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" is_pre_election: true
I20260809 06:23:07.889201 21022 raft_consensus.cc:2379] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b115279c79ef4ba5a46c1a103b77333f in current term 1: Already voted for candidate 047716f62295473687519bd142778f92 in this term.
I20260809 06:23:07.889871 21091 leader_election.cc:302] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b115279c79ef4ba5a46c1a103b77333f; no voters: 047716f62295473687519bd142778f92, f21b10e970bd4e4dbcde6eb37311a31c
I20260809 06:23:07.890332 21377 raft_consensus.cc:2726] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 06:23:07.892482 20868 catalog_manager.cc:5495] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 047716f62295473687519bd142778f92 (127.3.137.67). New cstate: current_term: 1 leader_uuid: "047716f62295473687519bd142778f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047716f62295473687519bd142778f92" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 } health_report { overall_health: UNKNOWN } } }
I20260809 06:23:07.910888 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.905136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.3.137.67:43811)
0809 06:23:07.905303 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.905310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.905507 (+ 197us) server_negotiation.cc:407] Connection header received
0809 06:23:07.905770 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.905775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.905850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.906012 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:07.906408 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.907253 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.908691 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.909118 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.909256 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.909792 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:07.910263 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:23:07.910421 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.910519 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.910601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 06:23:07.912971 21353 tablet.cc:2369] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:23:07.919142 21022 raft_consensus.cc:1261] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Refusing update from remote peer 047716f62295473687519bd142778f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:07.919281 21161 raft_consensus.cc:1261] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Refusing update from remote peer 047716f62295473687519bd142778f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:07.920099 21383 consensus_queue.cc:1035] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21b10e970bd4e4dbcde6eb37311a31c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:07.920434 21376 consensus_queue.cc:1035] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b115279c79ef4ba5a46c1a103b77333f" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:07.935635 21386 mvcc.cc:205] Tried to move back new op lower bound from 7316506984106917888 to 7316506983989133312. Current Snapshot: MvccSnapshot[applied={T|T < 7316506984106917888 or (T in {7316506984106917888})}]
I20260809 06:23:07.946244 20868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:07.946553 20868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:07.950295 20868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Sending DeleteTablet for 3 replicas of tablet bb8d32f9aedd4881b53d67096239e95d
I20260809 06:23:07.951515 21281 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8d32f9aedd4881b53d67096239e95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:55326
I20260809 06:23:07.951529 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8d32f9aedd4881b53d67096239e95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:59902
I20260809 06:23:07.951786 21001 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8d32f9aedd4881b53d67096239e95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:07 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:40490
I20260809 06:23:07.952729 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20912
I20260809 06:23:07.962632 21397 tablet_replica.cc:330] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: stopping tablet replica
I20260809 06:23:07.962736 21396 tablet_replica.cc:330] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: stopping tablet replica
I20260809 06:23:07.963316 21396 raft_consensus.cc:2229] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:07.963603 21397 raft_consensus.cc:2229] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:23:07.963886 21396 raft_consensus.cc:2258] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:07.964669 21399 tablet_replica.cc:330] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: stopping tablet replica
I20260809 06:23:07.964826 21397 raft_consensus.cc:2258] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:07.965171 21399 raft_consensus.cc:2229] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:07.965694 21399 raft_consensus.cc:2258] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:07.967469 21397 ts_tablet_manager.cc:1905] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:07.968261 21399 ts_tablet_manager.cc:1905] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:07.969264 21396 ts_tablet_manager.cc:1905] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:07.975123 21397 ts_tablet_manager.cc:1918] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:07.975382 21397 log.cc:1196] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/bb8d32f9aedd4881b53d67096239e95d
I20260809 06:23:07.975703 21399 ts_tablet_manager.cc:1918] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:07.975971 21399 log.cc:1196] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/bb8d32f9aedd4881b53d67096239e95d
I20260809 06:23:07.976079 21397 ts_tablet_manager.cc:1939] T bb8d32f9aedd4881b53d67096239e95d P 047716f62295473687519bd142778f92: Deleting consensus metadata
I20260809 06:23:07.976595 21399 ts_tablet_manager.cc:1939] T bb8d32f9aedd4881b53d67096239e95d P f21b10e970bd4e4dbcde6eb37311a31c: Deleting consensus metadata
I20260809 06:23:07.976965 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:07.953515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.3.137.65:45869)
0809 06:23:07.953697 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:07.953707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:07.954384 (+ 677us) server_negotiation.cc:407] Connection header received
0809 06:23:07.954631 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:07.954639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:07.954778 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:07.954907 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:07.957813 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.959500 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.962094 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:07.962829 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:07.962976 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:07.967470 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:07.967481 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:07.967498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:07.967546 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:07.971539 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.972415 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.972424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.972427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.972584 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:07.974949 (+ 2365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:07.974955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:07.974977 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:07.975518 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:07.975866 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:07.976264 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 06:23:07.976545 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:23:07.977692 21065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:07.978662 20852 catalog_manager.cc:4841] TS 047716f62295473687519bd142778f92 (127.3.137.67:43811): tablet bb8d32f9aedd4881b53d67096239e95d (table test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]) successfully deleted
I20260809 06:23:07.980693 21396 ts_tablet_manager.cc:1918] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:07.980966 21396 log.cc:1196] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/bb8d32f9aedd4881b53d67096239e95d
I20260809 06:23:07.981719 21396 ts_tablet_manager.cc:1939] T bb8d32f9aedd4881b53d67096239e95d P b115279c79ef4ba5a46c1a103b77333f: Deleting consensus metadata
I20260809 06:23:07.983701 20853 catalog_manager.cc:4841] TS b115279c79ef4ba5a46c1a103b77333f (127.3.137.66:41941): tablet bb8d32f9aedd4881b53d67096239e95d (table test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]) successfully deleted
I20260809 06:23:08.087935 20851 catalog_manager.cc:4841] TS f21b10e970bd4e4dbcde6eb37311a31c (127.3.137.65:45869): tablet bb8d32f9aedd4881b53d67096239e95d (table test-table [id=70c0f44aa1284d1e92c6eb600d8e4460]) successfully deleted
I20260809 06:23:08.088294 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20912
I20260809 06:23:08.109184 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 21074
I20260809 06:23:08.127337 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:08.109961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.3.137.66:41941)
0809 06:23:08.110136 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:08.110144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:08.110243 (+ 99us) server_negotiation.cc:407] Connection header received
0809 06:23:08.110460 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:08.110466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:08.110542 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:08.110647 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:08.112153 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:08.112887 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:08.114250 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:08.114654 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:08.114814 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:08.121349 (+ 6535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:08.121363 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:08.121378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:08.121409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:08.124331 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:08.125120 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:08.125128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:08.125130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:08.125275 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:08.125675 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:08.125681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:08.125684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:08.126058 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:08.126391 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:08.126746 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 06:23:08.126993 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:23:08.128121 21204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:08.222810 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 21074
I20260809 06:23:08.242663 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 21212
I20260809 06:23:08.258693 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:08.243559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.3.137.67:43811)
0809 06:23:08.243857 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:08.243865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:08.243916 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:23:08.244142 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:08.244148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:08.244230 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:08.244350 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:08.245823 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:08.246622 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:08.248122 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:08.248636 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:08.248781 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:08.251621 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:08.251637 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:08.251661 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:08.251701 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:08.255635 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:08.256368 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:08.256383 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:08.256385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:08.256507 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:08.257005 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:08.257010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:08.257014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:08.257465 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:08.257821 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:08.258115 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 06:23:08.258354 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260809 06:23:08.259567 21347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:08.382905 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 21212
I20260809 06:23:08.406689 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 20832
I20260809 06:23:08.407886 20904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:08.522256 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 20832
I20260809 06:23:28.548528 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--webserver_interface=127.3.137.126
--webserver_port=40123
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:28.882493 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:28.882768 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:28.882956 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:28.890225 21404 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:23:28.890343 21404 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:23:28.890413 21404 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:23:28.890470 21404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:28.890522 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:28.890587 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:28.890633 21404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:28.890679 21404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:28.900343 21404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=40123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:28.901492 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:28.902766 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:28.915432 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:28.915410 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:28.915377 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:28.921520 21404 server_base.cc:1034] running on GCE node
I20260809 06:23:28.922272 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:28.923728 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:28.924971 21404 hybrid_clock.cc:648] HybridClock initialized: now 1786256608924917 us; error 27 us; skew 500 ppm
I20260809 06:23:28.925256 21404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:28 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:28.928468 21404 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:23:28.929999 21404 webserver.cc:460] Webserver started at http://127.3.137.126:40123/ using document root <none> and password file <none>
I20260809 06:23:28.930694 21404 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:28.930838 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:28.936510 21404 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:23:28.939548 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:28.940852 21404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:23:28.941036 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "7659cbb5179f4131b19441e9012086dc"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "78796d4216d050dca987915291015c0d"
server_key_iv: "f83a410dcfe7591e68cda11badf0eefb"
server_key_version: "encryptionkey@0"
I20260809 06:23:28.942018 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:28.976362 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:28.980523 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:28.980813 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:29.000103 21404 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:45835
I20260809 06:23:29.000144 21476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:45835 every 8 connection(s)
I20260809 06:23:29.001520 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:29.008126 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21404
I20260809 06:23:29.008675 21477 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:23:29.009249 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:45869
--local_ip_for_outbound_sockets=127.3.137.65
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=37973
--webserver_interface=127.3.137.65
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:23:29.013746 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap starting.
I20260809 06:23:29.035799 21477 log.cc:824] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:29.055462 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:23:29.056007 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Bootstrap complete.
I20260809 06:23:29.065136 21477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:29.065986 21477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Initialized, Role: FOLLOWER
I20260809 06:23:29.066687 21477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:29.066931 21477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:29.067109 21477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:29.067314 21477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 2 FOLLOWER]: Advancing to term 3
I20260809 06:23:29.070902 21477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:29.071444 21477 leader_election.cc:302] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7659cbb5179f4131b19441e9012086dc; no voters:
I20260809 06:23:29.072115 21477 leader_election.cc:288] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 06:23:29.072258 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 3 FOLLOWER]: Leader election won for term 3
I20260809 06:23:29.073567 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [term 3 LEADER]: Becoming Leader. State: Replica: 7659cbb5179f4131b19441e9012086dc, State: Running, Role: LEADER
I20260809 06:23:29.074162 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } }
I20260809 06:23:29.074393 21477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:29.076776 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:23:29.077212 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:29.077337 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7659cbb5179f4131b19441e9012086dc. Latest consensus state: current_term: 3 leader_uuid: "7659cbb5179f4131b19441e9012086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7659cbb5179f4131b19441e9012086dc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 45835 } } }
I20260809 06:23:29.077663 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:29.080468 21487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:29.089517 21487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:29.091223 21487 catalog_manager.cc:1259] Loaded cluster ID: 6a1a10df1ed743588bcdf44877558a3d
I20260809 06:23:29.091406 21487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:29.095657 21487 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:29.097277 21487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Loaded TSK: 0
I20260809 06:23:29.097496 21487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7659cbb5179f4131b19441e9012086dc: Loaded TSK: 1
F20260809 06:23:29.098641 21487 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f97f27919ad google::LogMessage::Fail() at ??:0
@ 0x7f97f279488a google::LogMessage::SendToLog() at ??:0
@ 0x7f97f27914de google::LogMessage::Flush() at ??:0
@ 0x7f97f2792a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f97f64f57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
I20260809 06:23:29.113353 21499 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f97f64e473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f98081c01c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f98081bf783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f980824b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f980824b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f98082955e9 std::function<>::operator()() at ??:0
@ 0x7f98081c3cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f98081c1c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9808249610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9808249439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f97f53dd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f97f53e091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f97f53e0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f97f53bbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f97f86956db start_thread at ??:0
@ 0x7f97f162b61f clone at ??:0
W20260809 06:23:29.422644 21479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:29.422890 21479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:29.423031 21479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:29.430176 21479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:29.430280 21479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:29.430397 21479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:29.439883 21479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=37973
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:29.441015 21479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:29.442265 21479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:29.454998 21526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:29.455101 21527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:29.455065 21530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:29.456311 21479 server_base.cc:1034] running on GCE node
I20260809 06:23:29.456874 21479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:29.457728 21479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:29.458951 21479 hybrid_clock.cc:648] HybridClock initialized: now 1786256609458913 us; error 33 us; skew 500 ppm
I20260809 06:23:29.459286 21479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:29 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:29.462261 21479 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:23:29.463771 21479 webserver.cc:460] Webserver started at http://127.3.137.65:37973/ using document root <none> and password file <none>
I20260809 06:23:29.464396 21479 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:29.464540 21479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:29.469566 21479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 06:23:29.472165 21540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:29.473379 21479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:23:29.473549 21479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "f21b10e970bd4e4dbcde6eb37311a31c"
format_stamp: "Formatted at 2026-08-09 06:22:43 on dist-test-slave-2w3w"
server_key: "2fd51ec4549123531c907aff4b4cee4f"
server_key_iv: "327b922be5696bdf17e101409e1e9bbe"
server_key_version: "encryptionkey@0"
I20260809 06:23:29.474478 21479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:29.498840 21479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:29.502904 21479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:29.503201 21479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:29.504151 21479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:29.505453 21479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:29.505592 21479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:29.505713 21479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:29.505801 21479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:29.540385 21479 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:45869
I20260809 06:23:29.540431 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:45869 every 8 connection(s)
I20260809 06:23:29.541841 21479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:29.546446 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:29.544671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:58967)
0809 06:23:29.545373 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:29.545648 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
W20260809 06:23:29.547140 21661 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45835 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:45835: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
I20260809 06:23:29.547901 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:29.547608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:35403)
0809 06:23:29.547739 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:29.547803 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 06:23:29.548697 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:29.548409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:43669)
0809 06:23:29.548528 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:29.548598 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:23:29.548928 21661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:23:29.550904 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21479
I20260809 06:23:29.551587 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:41941
--local_ip_for_outbound_sockets=127.3.137.66
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=44649
--webserver_interface=127.3.137.66
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:29.882143 21664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:29.882437 21664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:29.882617 21664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:29.889806 21664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:29.889982 21664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:29.890158 21664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:29.899570 21664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:41941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=44649
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:29.900774 21664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:29.902050 21664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:29.915227 21669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:29.915134 21670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:29.915195 21672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:29.916303 21664 server_base.cc:1034] running on GCE node
I20260809 06:23:29.916814 21664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:29.917651 21664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:29.918861 21664 hybrid_clock.cc:648] HybridClock initialized: now 1786256609918818 us; error 68 us; skew 500 ppm
I20260809 06:23:29.919167 21664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:29 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256609, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:29.922174 21664 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:23:29.923556 21664 webserver.cc:460] Webserver started at http://127.3.137.66:44649/ using document root <none> and password file <none>
I20260809 06:23:29.924160 21664 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:29.924271 21664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:29.929400 21664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:23:29.931994 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:29.933231 21664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:23:29.933403 21664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "b115279c79ef4ba5a46c1a103b77333f"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "3166d32a8b850bef16f2f7dd87b1f09d"
server_key_iv: "acc639a804cf0362cb64c7a8accf0f49"
server_key_version: "encryptionkey@0"
I20260809 06:23:29.934351 21664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:29.969772 21664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:29.974437 21664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:29.974757 21664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:29.975822 21664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:29.977174 21664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:29.977294 21664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:29.977427 21664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:29.977523 21664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:30.012189 21664 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41941
I20260809 06:23:30.012251 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41941 every 8 connection(s)
I20260809 06:23:30.013684 21664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:30.018461 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.016645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:42325)
0809 06:23:30.017360 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.017604 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":282,"threads_started":1}
W20260809 06:23:30.019177 21802 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45835 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:45835: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
I20260809 06:23:30.019821 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21664
I20260809 06:23:30.020174 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.019797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:59169)
0809 06:23:30.019990 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.020064 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 06:23:30.020546 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:43811
--local_ip_for_outbound_sockets=127.3.137.67
--tserver_master_addrs=127.3.137.126:45835
--webserver_port=44619
--webserver_interface=127.3.137.67
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:23:30.021203 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.020825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:46871)
0809 06:23:30.020984 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.021064 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 06:23:30.021420 21802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:23:30.351418 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:30.351712 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:30.351889 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:30.359009 21805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:30.359149 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:30.359316 21805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:30.368760 21805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=44619
--tserver_master_addrs=127.3.137.126:45835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:30.369980 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:30.371263 21805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:30.384477 21810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:30.384443 21811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:30.384380 21813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:30.385106 21805 server_base.cc:1034] running on GCE node
I20260809 06:23:30.385685 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:30.386602 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:30.387828 21805 hybrid_clock.cc:648] HybridClock initialized: now 1786256610387789 us; error 48 us; skew 500 ppm
I20260809 06:23:30.388127 21805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:30 dist-test-slave-2w3w krb5kdc[20235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256610, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:30.391147 21805 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:23:30.392532 21805 webserver.cc:460] Webserver started at http://127.3.137.67:44619/ using document root <none> and password file <none>
I20260809 06:23:30.393170 21805 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:30.393316 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:30.398351 21805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:30.400880 21821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:30.402063 21805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 06:23:30.402235 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "047716f62295473687519bd142778f92"
format_stamp: "Formatted at 2026-08-09 06:22:44 on dist-test-slave-2w3w"
server_key: "895e73d9c8c9792d3884b75d3634257f"
server_key_iv: "94f489c3114078b8283a4d86b32b049b"
server_key_version: "encryptionkey@0"
I20260809 06:23:30.403272 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:30.417058 21805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:30.420648 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:30.420928 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:30.421823 21805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:30.423115 21805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:30.423237 21805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:30.423377 21805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:30.423466 21805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:30.456853 21805 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:43811
I20260809 06:23:30.456939 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:43811 every 8 connection(s)
I20260809 06:23:30.458272 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:30.458768 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 21805
I20260809 06:23:30.463037 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.461259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:48507)
0809 06:23:30.461951 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.462175 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
W20260809 06:23:30.463742 21944 heartbeater.cc:644] Failed to heartbeat to 127.3.137.126:45835 (0 consecutive failures): Network error: Failed to ping master at 127.3.137.126:45835: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
I20260809 06:23:30.464533 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.464238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:51411)
0809 06:23:30.464363 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.464428 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 06:23:30.465385 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.465063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:47447)
0809 06:23:30.465199 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.465280 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:23:30.465605 21944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:23:30.550205 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:30.549442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:46215)
0809 06:23:30.549885 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:30.549973 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260809 06:23:31.022845 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:31.021888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:47803)
0809 06:23:31.022388 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:31.022516 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":208,"threads_started":1}
I20260809 06:23:31.467048 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:31.466116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:34213)
0809 06:23:31.466631 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:31.466753 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260809 06:23:31.551800 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:31.551097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:52983)
0809 06:23:31.551562 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:31.551642 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260809 06:23:32.024473 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:32.023762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:58577)
0809 06:23:32.024231 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:32.024313 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":242,"threads_started":1}
I20260809 06:23:32.468564 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:32.467861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:57877)
0809 06:23:32.468309 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:32.468388 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260809 06:23:32.553359 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:32.552677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:47465)
0809 06:23:32.553108 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:32.553188 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260809 06:23:33.026118 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:33.025446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:40647)
0809 06:23:33.025880 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:33.025958 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260809 06:23:33.470029 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:33.469367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:33293)
0809 06:23:33.469789 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:33.469870 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
I20260809 06:23:33.554914 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:33.554195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:59587)
0809 06:23:33.554635 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:33.554727 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260809 06:23:34.027741 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:34.027071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:43385)
0809 06:23:34.027494 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:34.027573 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260809 06:23:34.471727 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:34.470933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:56543)
0809 06:23:34.471463 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:34.471543 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":257,"threads_started":1}
I20260809 06:23:34.556591 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:34.555863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:38113)
0809 06:23:34.556341 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:34.556424 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
I20260809 06:23:35.029426 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:35.028718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:44473)
0809 06:23:35.029170 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:35.029248 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
I20260809 06:23:35.473335 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:35.472603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:34003)
0809 06:23:35.473061 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:35.473144 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260809 06:23:35.558184 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:35.557488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:41479)
0809 06:23:35.557931 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:35.558023 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260809 06:23:36.031222 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:36.030362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:55097)
0809 06:23:36.030909 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:36.031062 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
I20260809 06:23:36.475103 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:36.474316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:50125)
0809 06:23:36.474788 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:36.474880 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260809 06:23:36.560009 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:36.559204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:52819)
0809 06:23:36.559699 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:36.559781 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":212,"threads_started":1}
I20260809 06:23:37.032897 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:37.032116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:58109)
0809 06:23:37.032605 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:37.032692 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":232,"threads_started":1}
I20260809 06:23:37.476823 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:37.476025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:51885)
0809 06:23:37.476522 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:37.476606 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":215,"threads_started":1}
I20260809 06:23:37.561664 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:37.560977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:37501)
0809 06:23:37.561409 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:37.561506 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260809 06:23:38.034662 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:38.033870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:48199)
0809 06:23:38.034420 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:38.034499 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":316,"threads_started":1}
I20260809 06:23:38.478571 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:38.477811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:45027)
0809 06:23:38.478261 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:38.478352 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260809 06:23:38.563359 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:38.562536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:54729)
0809 06:23:38.563111 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:38.563190 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260809 06:23:39.036527 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:39.035633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:33835)
0809 06:23:39.036177 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:39.036333 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":243,"threads_started":1}
I20260809 06:23:39.480281 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:39.479471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:59805)
0809 06:23:39.480006 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:39.480095 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
I20260809 06:23:39.564992 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:39.564281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:35781)
0809 06:23:39.564738 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:39.564821 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260809 06:23:40.038264 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:40.037513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:50877)
0809 06:23:40.037982 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:40.038064 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260809 06:23:40.481897 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:40.481227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:39371)
0809 06:23:40.481665 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:40.481745 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
I20260809 06:23:40.566655 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:40.565933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:34675)
0809 06:23:40.566377 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:40.566471 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":210,"threads_started":1}
I20260809 06:23:41.039865 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:41.039140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:46357)
0809 06:23:41.039592 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:41.039684 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":229,"threads_started":1}
I20260809 06:23:41.483590 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:41.482841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:48955)
0809 06:23:41.483322 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:41.483406 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":286,"threads_started":1}
I20260809 06:23:41.568351 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:41.567608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:45379)
0809 06:23:41.568112 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:41.568194 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260809 06:23:42.041481 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:42.040804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:51841)
0809 06:23:42.041243 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:42.041327 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":244,"threads_started":1}
I20260809 06:23:42.485199 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:42.484503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:36913)
0809 06:23:42.484970 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:42.485049 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260809 06:23:42.569967 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:42.569206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:47147)
0809 06:23:42.569660 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:42.569781 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
I20260809 06:23:43.043151 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:43.042309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:36271)
0809 06:23:43.042793 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:43.042894 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
I20260809 06:23:43.486863 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:43.486065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:38841)
0809 06:23:43.486575 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:43.486671 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":262,"threads_started":1}
I20260809 06:23:43.571756 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:43.570912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:57309)
0809 06:23:43.571491 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:43.571575 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":347,"threads_started":1}
I20260809 06:23:44.044687 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:44.043954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:52177)
0809 06:23:44.044382 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:44.044495 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
I20260809 06:23:44.488476 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:44.487745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:37845)
0809 06:23:44.488220 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:44.488303 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260809 06:23:44.573522 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:44.572803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:58707)
0809 06:23:44.573235 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:44.573344 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260809 06:23:45.047189 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.046074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.66:57795)
0809 06:23:45.046705 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:45.046844 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
I20260809 06:23:45.471542 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 21479
I20260809 06:23:45.490403 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.489600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.67:44603)
0809 06:23:45.490104 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:45.490195 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260809 06:23:45.493291 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.472745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.3.137.65:45869)
0809 06:23:45.473374 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:45.473388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:45.473427 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:23:45.474978 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:45.474989 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:45.475413 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:45.475786 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:45.477396 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.478903 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.480326 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.481030 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.481162 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:45.486927 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:45.486935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:45.486950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:45.487037 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:45.490278 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.491086 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.491092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.491098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.491259 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.491612 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.491615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.491618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.492120 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:45.492355 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:45.492689 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 06:23:45.493035 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260809 06:23:45.494621 21656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:45.584722 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.583418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:45835 (local address 127.3.137.65:39743)
0809 06:23:45.584074 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:45.584317 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.126:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":352,"threads_started":1}
I20260809 06:23:45.584729 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 21479
I20260809 06:23:45.604656 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 21664
I20260809 06:23:45.624099 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.605715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.3.137.66:41941)
0809 06:23:45.606295 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:45.606310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:45.606346 (+ 36us) server_negotiation.cc:407] Connection header received
0809 06:23:45.608134 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:45.608144 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:45.608561 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:45.608859 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:45.610204 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.611573 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.612798 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.613433 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.613562 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:45.616390 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:45.616402 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:45.616423 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:45.616493 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:45.619901 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.620859 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.620867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.620876 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.621098 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.621596 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.621602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.621606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.622466 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:45.622816 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:45.623379 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 06:23:45.623745 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260809 06:23:45.625741 21797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:45.712913 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 21664
I20260809 06:23:45.733596 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 21805
I20260809 06:23:45.751524 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:45.734764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.3.137.67:43811)
0809 06:23:45.735463 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:45.735478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:45.735520 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:23:45.737205 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:45.737214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:45.737636 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:45.737918 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:45.739366 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.740564 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.742082 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:45.742726 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:45.742864 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:45.745630 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:45.745637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:45.745651 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:45.745685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:45.748524 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.749192 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.749198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.749204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.749374 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:45.749683 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:45.749686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:45.749688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:45.750155 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:45.750383 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:45.750728 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 06:23:45.751192 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":384,"threads_started":1}
I20260809 06:23:45.752916 21939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:45.837723 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 21805
2026-08-09T06:23:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64488 ms)
[----------] 30 tests from SecurityITest (301432 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:23:45 dist-test-slave-2w3w krb5kdc[22005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:23:45 dist-test-slave-2w3w krb5kdc[22005](info): set up 2 sockets
Aug 09 06:23:45 dist-test-slave-2w3w krb5kdc[22005](info): commencing operation
krb5kdc: starting...
W20260809 06:23:47.017836 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:23:47 dist-test-slave-2w3w krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:23:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:23:47.311781 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43077
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:43077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:47.650138 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:47.650472 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:47.650645 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:47.657755 22022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:47.657931 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:47.658030 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:47.658133 22022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:47.658215 22022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:47.667863 22022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:43077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:47.669122 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:47.670405 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:47.683300 22027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:47.683336 22030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:47.683247 22028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:47.684352 22022 server_base.cc:1034] running on GCE node
I20260809 06:23:47.685133 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:47.686568 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:47.687795 22022 hybrid_clock.cc:648] HybridClock initialized: now 1786256627687746 us; error 81 us; skew 500 ppm
I20260809 06:23:47.688146 22022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:47 dist-test-slave-2w3w krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:47.691326 22022 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:23:47.692862 22022 webserver.cc:460] Webserver started at http://127.3.137.126:35835/ using document root <none> and password file <none>
I20260809 06:23:47.693542 22022 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:47.693686 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:47.693987 22022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:47.697026 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "0a74ff86cda8413ea5cfc79af04c6374"
format_stamp: "Formatted at 2026-08-09 06:23:47 on dist-test-slave-2w3w"
server_key: "75829a9287adb564d3e92ce767863485"
server_key_iv: "317155af2e14859d81eda3efbdfc232d"
server_key_version: "encryptionkey@0"
I20260809 06:23:47.697783 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "0a74ff86cda8413ea5cfc79af04c6374"
format_stamp: "Formatted at 2026-08-09 06:23:47 on dist-test-slave-2w3w"
server_key: "75829a9287adb564d3e92ce767863485"
server_key_iv: "317155af2e14859d81eda3efbdfc232d"
server_key_version: "encryptionkey@0"
I20260809 06:23:47.702920 22022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:23:47.706321 22039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:47.707659 22022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:23:47.707836 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "0a74ff86cda8413ea5cfc79af04c6374"
format_stamp: "Formatted at 2026-08-09 06:23:47 on dist-test-slave-2w3w"
server_key: "75829a9287adb564d3e92ce767863485"
server_key_iv: "317155af2e14859d81eda3efbdfc232d"
server_key_version: "encryptionkey@0"
I20260809 06:23:47.708029 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:47.730523 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:47.734277 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:47.734570 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:47.753360 22022 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:43077
I20260809 06:23:47.753383 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:43077 every 8 connection(s)
I20260809 06:23:47.754686 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:47.758260 22095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:47.759651 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22022
I20260809 06:23:47.759857 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:23:47.760612 3621 external_mini_cluster.cc:1428] Setting key 5fa8b0b8ad879f4ef9c306cd4dac1eaf
I20260809 06:23:47.769989 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: Bootstrap starting.
I20260809 06:23:47.773699 22095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:47.775060 22095 log.cc:824] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:23:47 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:47.777889 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: No bootstrap required, opened a new log
I20260809 06:23:47.784123 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:47.762540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.3.137.126:43077)
0809 06:23:47.763364 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:47.763378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:47.763418 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:23:47.765088 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:47.765107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:47.765458 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:47.765794 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:47.767653 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:47.768956 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:47.770837 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:47.771539 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:47.771670 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:47.776884 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:47.776891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:47.776904 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:47.776943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:47.780131 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:47.780862 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:47.780869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:47.780874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:47.781035 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:47.781446 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:47.781452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:47.781455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:47.781953 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:47.782273 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:47.782717 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 06:23:47.783177 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":408,"threads_started":1}
I20260809 06:23:47.785789 22095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } }
I20260809 06:23:47.786155 22095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:47.786280 22095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a74ff86cda8413ea5cfc79af04c6374, State: Initialized, Role: FOLLOWER
I20260809 06:23:47.786875 22095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } }
I20260809 06:23:47.787175 22095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:47.787322 22095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:47.787505 22095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:47.789609 22095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } }
I20260809 06:23:47.790068 22095 leader_election.cc:302] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a74ff86cda8413ea5cfc79af04c6374; no voters:
I20260809 06:23:47.790611 22095 leader_election.cc:288] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:23:47.790745 22101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:47.791473 22101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [term 1 LEADER]: Becoming Leader. State: Replica: 0a74ff86cda8413ea5cfc79af04c6374, State: Running, Role: LEADER
I20260809 06:23:47.791975 22101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } }
I20260809 06:23:47.792598 22095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:47.795071 22103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a74ff86cda8413ea5cfc79af04c6374. Latest consensus state: current_term: 1 leader_uuid: "0a74ff86cda8413ea5cfc79af04c6374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } } }
I20260809 06:23:47.795462 22103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:47.795958 22102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a74ff86cda8413ea5cfc79af04c6374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a74ff86cda8413ea5cfc79af04c6374" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 43077 } } }
I20260809 06:23:47.796416 22102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374 [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:47.797838 22107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:47.803565 22107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:47.812943 22107 catalog_manager.cc:1347] Generated new cluster ID: 7d5f41b94be3463db204c6e1969053e2
I20260809 06:23:47.813059 22107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:47.825721 22107 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:23:47.827418 22107 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:47.835357 22107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: Generated new TSK 0
I20260809 06:23:47.836561 22107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:23:47.918783 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:48.260679 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:48.261008 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:48.261197 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:48.268348 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:48.268498 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:48.268709 22125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:48.279161 22125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:48.280441 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:48.281710 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:48.295044 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:48.295074 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:48.295097 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:48.295768 22125 server_base.cc:1034] running on GCE node
I20260809 06:23:48.296499 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:48.297530 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:48.298799 22125 hybrid_clock.cc:648] HybridClock initialized: now 1786256628298717 us; error 93 us; skew 500 ppm
I20260809 06:23:48.299206 22125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:48 dist-test-slave-2w3w krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:48.302472 22125 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:23:48.303915 22125 webserver.cc:460] Webserver started at http://127.3.137.65:37629/ using document root <none> and password file <none>
I20260809 06:23:48.304561 22125 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:48.304663 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:48.304934 22125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:48.307848 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "e2622434b9964e6d92005467e9a72893"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "5acd5b643b7261cf7f4b5d401e8704c6"
server_key_iv: "7208df85ddfc44fe0ceeda69e9a8abdb"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.308589 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "e2622434b9964e6d92005467e9a72893"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "5acd5b643b7261cf7f4b5d401e8704c6"
server_key_iv: "7208df85ddfc44fe0ceeda69e9a8abdb"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.313724 22125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:23:48.317097 22140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.318365 22125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:48.318521 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "e2622434b9964e6d92005467e9a72893"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "5acd5b643b7261cf7f4b5d401e8704c6"
server_key_iv: "7208df85ddfc44fe0ceeda69e9a8abdb"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.318702 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:48.339066 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:48.343091 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:48.343389 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:48.344388 22125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:48.345715 22125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:48.345817 22125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.345965 22125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:48.346027 22125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.380280 22125 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:45929
I20260809 06:23:48.380333 22264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:45929 every 8 connection(s)
I20260809 06:23:48.381633 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:48.390141 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22125
I20260809 06:23:48.390381 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:23:48.391113 3621 external_mini_cluster.cc:1428] Setting key 70e7714e11584be55561776a34ad2eec
Aug 09 06:23:48 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:48.402770 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:48.384238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:57029 (local address 127.3.137.126:43077)
0809 06:23:48.384679 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:48.384684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:48.385606 (+ 922us) server_negotiation.cc:407] Connection header received
0809 06:23:48.387539 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:48.387546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:48.387636 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:48.387761 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:48.389235 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.390001 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.391824 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.392396 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.392513 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:48.396660 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:48.396665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:48.396668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:48.396700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:48.399538 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:48.400492 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:48.400496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:48.400498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:48.400623 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:48.401105 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:48.401108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:48.401110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:48.401355 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:48.401479 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:48.402335 (+ 856us) server_negotiation.cc:299] Negotiation successful
0809 06:23:48.402517 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260809 06:23:48.404302 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:48.384757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43077 (local address 127.3.137.65:57029)
0809 06:23:48.385446 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:48.385491 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:48.387257 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:48.388002 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:48.388010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:48.388511 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:48.389053 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:48.389071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.390176 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.390180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:48.391635 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:48.391672 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.392633 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.392641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:48.392924 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:48.393683 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:48.393706 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:48.396438 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:48.399687 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:48.399698 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:48.399715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:48.400360 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:48.400732 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:48.400739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:48.400745 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:48.400994 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:48.401480 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:48.401490 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:48.401927 (+ 437us) client_negotiation.cc:769] Sending connection context
0809 06:23:48.402241 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 06:23:48.402822 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":269,"threads_started":1}
I20260809 06:23:48.405884 22265 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43077
I20260809 06:23:48.406195 22265 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:48.407131 22265 heartbeater.cc:511] Master 127.3.137.126:43077 requested a full tablet report, sending...
I20260809 06:23:48.409427 22056 ts_manager.cc:194] Registered new tserver with Master: e2622434b9964e6d92005467e9a72893 (127.3.137.65:45929)
I20260809 06:23:48.411021 22056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:57029
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:23:48.458171 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:48.789805 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:48.790086 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:48.790179 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:48.797410 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:48.797515 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:48.797627 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:48.807410 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:48.808622 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:48.809844 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:48.822942 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:48.822919 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:48.823159 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:48.828332 22271 server_base.cc:1034] running on GCE node
I20260809 06:23:48.829042 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:48.830041 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:48.831271 22271 hybrid_clock.cc:648] HybridClock initialized: now 1786256628831228 us; error 57 us; skew 500 ppm
I20260809 06:23:48.831591 22271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:48 dist-test-slave-2w3w krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:48.834784 22271 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:23:48.836408 22271 webserver.cc:460] Webserver started at http://127.3.137.66:38565/ using document root <none> and password file <none>
I20260809 06:23:48.837136 22271 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:48.837288 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:48.837606 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:48.840603 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "3687c9265c6e4a6ea8b792a0005d39c3"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "d043a1907e4386e2c71f429e30a99807"
server_key_iv: "cb2adff4cc681294b3b41f6edb811dc0"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.841408 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "3687c9265c6e4a6ea8b792a0005d39c3"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "d043a1907e4386e2c71f429e30a99807"
server_key_iv: "cb2adff4cc681294b3b41f6edb811dc0"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.846477 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:23:48.849994 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.851274 22271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:23:48.851449 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "3687c9265c6e4a6ea8b792a0005d39c3"
format_stamp: "Formatted at 2026-08-09 06:23:48 on dist-test-slave-2w3w"
server_key: "d043a1907e4386e2c71f429e30a99807"
server_key_iv: "cb2adff4cc681294b3b41f6edb811dc0"
server_key_version: "encryptionkey@0"
I20260809 06:23:48.851656 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:48.865919 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:48.870038 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:48.870332 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:48.871351 22271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:48.872632 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:48.872742 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.872896 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:48.872983 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:48.907428 22271 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:36555
I20260809 06:23:48.907470 22413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:36555 every 8 connection(s)
I20260809 06:23:48.908772 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:48.916056 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22271
I20260809 06:23:48.916260 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:23:48.916849 3621 external_mini_cluster.cc:1428] Setting key fa698bba5469acc8ed3568b41a83b22d
Aug 09 06:23:48 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:48.931190 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:48.911613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:37151 (local address 127.3.137.126:43077)
0809 06:23:48.912186 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:48.912193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:48.913030 (+ 837us) server_negotiation.cc:407] Connection header received
0809 06:23:48.914991 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:48.914997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:48.915088 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:48.915226 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:48.917303 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.918229 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.920591 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.921057 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.921156 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:48.925398 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:48.925403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:48.925406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:48.925438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:48.928052 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:48.928797 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:48.928809 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:48.928813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:48.928959 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:48.929400 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:48.929406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:48.929409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:48.929681 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:48.929842 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:48.930536 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 06:23:48.930767 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":315,"threads_started":1}
I20260809 06:23:48.932219 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:48.912056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43077 (local address 127.3.137.66:37151)
0809 06:23:48.912866 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:48.912907 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:48.914677 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:48.915474 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:48.915482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:48.916282 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:48.917093 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:48.917119 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.918400 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.918404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:48.920415 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:48.920435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:48.921291 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:48.921300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:48.921644 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:48.922392 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:48.922450 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:48.925155 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:48.928211 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:48.928219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:48.928237 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:48.928660 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:48.929076 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:48.929081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:48.929085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:48.929281 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:48.929813 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:48.929820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:48.930169 (+ 349us) client_negotiation.cc:769] Sending connection context
0809 06:23:48.930449 (+ 280us) client_negotiation.cc:240] Negotiation successful
0809 06:23:48.931032 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":330,"threads_started":1}
I20260809 06:23:48.934027 22414 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43077
I20260809 06:23:48.934396 22414 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:48.935353 22414 heartbeater.cc:511] Master 127.3.137.126:43077 requested a full tablet report, sending...
I20260809 06:23:48.937063 22056 ts_manager.cc:194] Registered new tserver with Master: 3687c9265c6e4a6ea8b792a0005d39c3 (127.3.137.66:36555)
I20260809 06:23:48.937909 22056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:37151
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:23:48.985531 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:49.320490 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:49.320776 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:49.320953 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:49.328245 22420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:49.328367 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:49.328553 22420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:49.338066 22420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:49.339308 22420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:49.340615 22420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:49.353940 22428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:49.354173 22425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:49.353981 22426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:49.355139 22420 server_base.cc:1034] running on GCE node
I20260809 06:23:49.355695 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:49.356686 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:49.357934 22420 hybrid_clock.cc:648] HybridClock initialized: now 1786256629357887 us; error 56 us; skew 500 ppm
I20260809 06:23:49.358249 22420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:49 dist-test-slave-2w3w krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:49.361334 22420 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:23:49.362784 22420 webserver.cc:460] Webserver started at http://127.3.137.67:45837/ using document root <none> and password file <none>
I20260809 06:23:49.363556 22420 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:49.363708 22420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:49.364032 22420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:49.366889 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "b3fc9fff896642cfa6796e14c2bdd50a"
format_stamp: "Formatted at 2026-08-09 06:23:49 on dist-test-slave-2w3w"
server_key: "96a844ee8c1200381f9e8dbb02818820"
server_key_iv: "4544bd16119045975f85cd76d740631a"
server_key_version: "encryptionkey@0"
I20260809 06:23:49.367730 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "b3fc9fff896642cfa6796e14c2bdd50a"
format_stamp: "Formatted at 2026-08-09 06:23:49 on dist-test-slave-2w3w"
server_key: "96a844ee8c1200381f9e8dbb02818820"
server_key_iv: "4544bd16119045975f85cd76d740631a"
server_key_version: "encryptionkey@0"
I20260809 06:23:49.373221 22420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:23:49.376972 22435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:49.378280 22420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:49.378472 22420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "b3fc9fff896642cfa6796e14c2bdd50a"
format_stamp: "Formatted at 2026-08-09 06:23:49 on dist-test-slave-2w3w"
server_key: "96a844ee8c1200381f9e8dbb02818820"
server_key_iv: "4544bd16119045975f85cd76d740631a"
server_key_version: "encryptionkey@0"
I20260809 06:23:49.378697 22420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:49.396180 22420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:49.400619 22420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:49.400940 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:49.401960 22420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:49.403429 22420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:49.403544 22420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:49.403724 22420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:49.403820 22420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:49.414794 22265 heartbeater.cc:503] Master 127.3.137.126:43077 was elected leader, sending a full tablet report...
I20260809 06:23:49.441491 22420 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:35399
I20260809 06:23:49.441536 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:35399 every 8 connection(s)
I20260809 06:23:49.442922 22420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:49.444844 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22420
I20260809 06:23:49.445034 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:23:49.445552 3621 external_mini_cluster.cc:1428] Setting key bc826ec4a6382a1235b4a79128aba20a
Aug 09 06:23:49 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:49.464512 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.445974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:33507 (local address 127.3.137.126:43077)
0809 06:23:49.446512 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.446519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.447626 (+ 1107us) server_negotiation.cc:407] Connection header received
0809 06:23:49.449947 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.449960 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.450037 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.450130 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:49.451771 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.452560 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.454222 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.454603 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.454692 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.458347 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:49.458352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:49.458355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:49.458386 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:49.461710 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.462434 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.462438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.462440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.462563 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.463010 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.463016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.463019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.463286 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:49.463410 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.464082 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.464243 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":375,"threads_started":1}
I20260809 06:23:49.465610 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.446514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:43077 (local address 127.3.137.67:33507)
0809 06:23:49.447461 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.447509 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.449683 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.450352 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.450360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.450937 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:49.451614 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.451634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.452705 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.452709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.454095 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.454107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.454830 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.454836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.455115 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.455812 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:49.455835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:49.458147 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:49.461836 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:49.461843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:49.461856 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:49.462322 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:49.462659 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:49.462663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:49.462665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:49.462850 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:49.463408 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:49.463414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:49.463748 (+ 334us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.463978 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.464377 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":423,"threads_started":1}
I20260809 06:23:49.467042 22559 heartbeater.cc:347] Connected to a master server at 127.3.137.126:43077
I20260809 06:23:49.467337 22559 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:49.468077 22559 heartbeater.cc:511] Master 127.3.137.126:43077 requested a full tablet report, sending...
I20260809 06:23:49.469734 22056 ts_manager.cc:194] Registered new tserver with Master: b3fc9fff896642cfa6796e14c2bdd50a (127.3.137.67:35399)
I20260809 06:23:49.470592 22056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:33507
I20260809 06:23:49.473637 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:23:49.491544 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.478623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.3.137.126:43077)
0809 06:23:49.478793 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.478798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.479343 (+ 545us) server_negotiation.cc:407] Connection header received
0809 06:23:49.479437 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.479441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.479527 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.479610 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:49.480979 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.481693 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.483294 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.483746 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.483842 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.486294 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:49.486298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:49.486302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:49.486329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:49.488834 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.489514 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.489519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.489522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.489667 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.490035 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.490039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.490041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.490345 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:49.490556 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.490927 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.491157 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:23:49.495594 22056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:23:49.502676 22056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:23:49.537652 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.515054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.3.137.67:35399)
0809 06:23:49.529747 (+ 14693us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.529755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.529802 (+ 47us) server_negotiation.cc:407] Connection header received
0809 06:23:49.529905 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.529915 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.530109 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.531045 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:49.531343 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.532893 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.535338 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.536448 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.536596 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.536835 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.537120 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.537291 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14524,"thread_start_us":14370,"threads_started":1}
I20260809 06:23:49.537762 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.514740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:35399 (local address 127.0.0.1:33470)
0809 06:23:49.521687 (+ 6947us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.521743 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.521977 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.530346 (+ 8369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.530350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.530375 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:49.530827 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.530844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.533039 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.533047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.535127 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.535142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.536654 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.536660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.536874 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.536929 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.537070 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.537269 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6622,"mutex_wait_us":258,"thread_start_us":415,"threads_started":1}
I20260809 06:23:49.538121 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.515943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:36555 (local address 127.0.0.1:35954)
0809 06:23:49.528364 (+ 12421us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.528381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.528490 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.529565 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.529572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.529602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:49.529999 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.530015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.532315 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.532322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.535298 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.535313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.537211 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.537217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.537393 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.537434 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.537510 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.537650 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12294,"mutex_wait_us":156,"thread_start_us":12125,"threads_started":1}
I20260809 06:23:49.538697 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.515657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45929 (local address 127.0.0.1:56326)
0809 06:23:49.527995 (+ 12338us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.528019 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.528191 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.529920 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.529926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.529967 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:49.530498 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.530509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.532332 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.532337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.535669 (+ 3332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.535683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.537821 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.537828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.538024 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.538075 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.538177 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.538317 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12163,"mutex_wait_us":129,"thread_start_us":11983,"threads_started":1}
I20260809 06:23:49.539283 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.515834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.3.137.65:45929)
0809 06:23:49.529329 (+ 13495us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.529341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.529372 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:23:49.529482 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.529491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.529699 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.529926 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:49.530691 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.532185 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.535848 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.537309 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.537438 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.537663 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.538379 (+ 716us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.538547 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13298,"thread_start_us":13105,"threads_started":1}
I20260809 06:23:49.539583 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.515701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.3.137.66:36555)
0809 06:23:49.528787 (+ 13086us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.528800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.528832 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:23:49.528945 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.528954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.529258 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.530214 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:49.530471 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.532141 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.535491 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.536953 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.538709 (+ 1756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.538912 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.539008 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.539188 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12867,"thread_start_us":12883,"threads_started":1}
I20260809 06:23:49.541713 22486 tablet_service.cc:1467] Processing CreateTablet for tablet 178da8f230c240b0b42cee748f3a13d1 (DEFAULT_TABLE table=test-table [id=fe341efbcce0467bbd126f58c79ad8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:49.542441 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 178da8f230c240b0b42cee748f3a13d1 (DEFAULT_TABLE table=test-table [id=fe341efbcce0467bbd126f58c79ad8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:49.542918 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178da8f230c240b0b42cee748f3a13d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:49.543730 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178da8f230c240b0b42cee748f3a13d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:49.546207 22195 tablet_service.cc:1467] Processing CreateTablet for tablet 178da8f230c240b0b42cee748f3a13d1 (DEFAULT_TABLE table=test-table [id=fe341efbcce0467bbd126f58c79ad8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:49.547477 22195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178da8f230c240b0b42cee748f3a13d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:49.554800 22578 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Bootstrap starting.
I20260809 06:23:49.558239 22578 tablet_bootstrap.cc:654] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:49.559347 22578 log.cc:824] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:49.561942 22580 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Bootstrap starting.
I20260809 06:23:49.562243 22578 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: No bootstrap required, opened a new log
I20260809 06:23:49.562565 22578 ts_tablet_manager.cc:1397] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:23:49.562860 22581 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Bootstrap starting.
I20260809 06:23:49.565523 22580 tablet_bootstrap.cc:654] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:49.566543 22580 log.cc:824] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:49.567569 22581 tablet_bootstrap.cc:654] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:49.568799 22581 log.cc:824] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:49.569145 22580 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: No bootstrap required, opened a new log
I20260809 06:23:49.569458 22580 ts_tablet_manager.cc:1397] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260809 06:23:49.571858 22581 tablet_bootstrap.cc:492] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: No bootstrap required, opened a new log
I20260809 06:23:49.572172 22581 ts_tablet_manager.cc:1397] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 06:23:49.573132 22578 raft_consensus.cc:348] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.573614 22578 raft_consensus.cc:374] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:49.573717 22578 raft_consensus.cc:729] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fc9fff896642cfa6796e14c2bdd50a, State: Initialized, Role: FOLLOWER
I20260809 06:23:49.574333 22578 consensus_queue.cc:260] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.575508 22559 heartbeater.cc:503] Master 127.3.137.126:43077 was elected leader, sending a full tablet report...
I20260809 06:23:49.576148 22578 ts_tablet_manager.cc:1428] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260809 06:23:49.577713 22580 raft_consensus.cc:348] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.578135 22580 raft_consensus.cc:374] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:49.578243 22580 raft_consensus.cc:729] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3687c9265c6e4a6ea8b792a0005d39c3, State: Initialized, Role: FOLLOWER
I20260809 06:23:49.578843 22580 consensus_queue.cc:260] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.579980 22414 heartbeater.cc:503] Master 127.3.137.126:43077 was elected leader, sending a full tablet report...
I20260809 06:23:49.580466 22580 ts_tablet_manager.cc:1428] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260809 06:23:49.584020 22581 raft_consensus.cc:348] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.584470 22581 raft_consensus.cc:374] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:49.584578 22581 raft_consensus.cc:729] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2622434b9964e6d92005467e9a72893, State: Initialized, Role: FOLLOWER
I20260809 06:23:49.585067 22581 consensus_queue.cc:260] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.586580 22581 ts_tablet_manager.cc:1428] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260809 06:23:49.634034 22266 tablet.cc:2369] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:23:49.660866 22415 tablet.cc:2369] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:23:49.673462 22586 raft_consensus.cc:482] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:49.673786 22586 raft_consensus.cc:504] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.675513 22586 leader_election.cc:288] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3fc9fff896642cfa6796e14c2bdd50a (127.3.137.67:35399), e2622434b9964e6d92005467e9a72893 (127.3.137.65:45929)
I20260809 06:23:49.683454 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.676129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:45171 (local address 127.3.137.67:35399)
0809 06:23:49.676389 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.676398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.677275 (+ 877us) server_negotiation.cc:407] Connection header received
0809 06:23:49.677473 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.677477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.677537 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.677640 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:49.678213 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.679033 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.681307 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.682413 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.682533 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.682585 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.683010 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.683148 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:23:49.683653 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.676055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:35399 (local address 127.3.137.66:45171)
0809 06:23:49.677157 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.677173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.677310 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.677655 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.677660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.677684 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:49.678065 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.678073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.679175 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.679181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.681134 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.681146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.682626 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.682634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.682801 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.682848 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.682947 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.683154 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":53,"thread_start_us":769,"threads_started":1}
I20260809 06:23:49.683909 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.676143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:40317 (local address 127.3.137.65:45929)
0809 06:23:49.676401 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.676411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.677374 (+ 963us) server_negotiation.cc:407] Connection header received
0809 06:23:49.677585 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.677590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.677660 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.677759 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:49.678417 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.679481 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.681731 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.682743 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.682902 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.683028 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.683514 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.683653 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:23:49.684561 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.676098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:45929 (local address 127.3.137.66:40317)
0809 06:23:49.677258 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:49.677277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:49.677420 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:49.677777 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:49.677781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:49.677820 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:49.678267 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.678276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.679629 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.679634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.681570 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:49.681584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.683039 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.683048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:49.683247 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.683287 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 06:23:49.684114 (+ 827us) client_negotiation.cc:240] Negotiation successful
0809 06:23:49.684221 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":1,"thread_start_us":677,"threads_started":1}
I20260809 06:23:49.685751 22218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178da8f230c240b0b42cee748f3a13d1" candidate_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2622434b9964e6d92005467e9a72893" is_pre_election: true
I20260809 06:23:49.685886 22509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178da8f230c240b0b42cee748f3a13d1" candidate_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" is_pre_election: true
I20260809 06:23:49.686393 22218 raft_consensus.cc:2454] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3687c9265c6e4a6ea8b792a0005d39c3 in term 0.
I20260809 06:23:49.686460 22509 raft_consensus.cc:2454] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3687c9265c6e4a6ea8b792a0005d39c3 in term 0.
I20260809 06:23:49.687179 22292 leader_election.cc:302] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3687c9265c6e4a6ea8b792a0005d39c3, b3fc9fff896642cfa6796e14c2bdd50a, e2622434b9964e6d92005467e9a72893; no voters:
I20260809 06:23:49.687626 22586 raft_consensus.cc:2781] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:23:49.687783 22586 raft_consensus.cc:482] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:23:49.687911 22586 raft_consensus.cc:3037] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:49.690088 22586 raft_consensus.cc:504] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.690729 22586 leader_election.cc:288] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [CANDIDATE]: Term 1 election: Requested vote from peers b3fc9fff896642cfa6796e14c2bdd50a (127.3.137.67:35399), e2622434b9964e6d92005467e9a72893 (127.3.137.65:45929)
I20260809 06:23:49.691210 22509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178da8f230c240b0b42cee748f3a13d1" candidate_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fc9fff896642cfa6796e14c2bdd50a"
I20260809 06:23:49.691222 22218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178da8f230c240b0b42cee748f3a13d1" candidate_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2622434b9964e6d92005467e9a72893"
I20260809 06:23:49.691422 22509 raft_consensus.cc:3037] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:49.691434 22218 raft_consensus.cc:3037] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:49.693935 22509 raft_consensus.cc:2454] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3687c9265c6e4a6ea8b792a0005d39c3 in term 1.
I20260809 06:23:49.694028 22218 raft_consensus.cc:2454] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3687c9265c6e4a6ea8b792a0005d39c3 in term 1.
I20260809 06:23:49.694339 22292 leader_election.cc:302] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3687c9265c6e4a6ea8b792a0005d39c3, b3fc9fff896642cfa6796e14c2bdd50a; no voters:
I20260809 06:23:49.694689 22586 raft_consensus.cc:2781] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 06:23:49.694737 22560 tablet.cc:2369] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:23:49.695319 22586 raft_consensus.cc:686] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3687c9265c6e4a6ea8b792a0005d39c3, State: Running, Role: LEADER
I20260809 06:23:49.695789 22586 consensus_queue.cc:237] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } }
I20260809 06:23:49.700636 22056 catalog_manager.cc:5495] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3687c9265c6e4a6ea8b792a0005d39c3 (127.3.137.66). New cstate: current_term: 1 leader_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3687c9265c6e4a6ea8b792a0005d39c3" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 36555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 } health_report { overall_health: UNKNOWN } } }
I20260809 06:23:49.728811 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.723691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.3.137.66:36555)
0809 06:23:49.723889 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.723893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.723911 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:23:49.724095 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.724098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.724176 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.724304 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:49.724651 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.725383 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.726787 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.727257 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.727355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.727787 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:49.728297 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:23:49.728408 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.728518 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.728584 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:23:49.736898 22509 raft_consensus.cc:1261] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 1 FOLLOWER]: Refusing update from remote peer 3687c9265c6e4a6ea8b792a0005d39c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:49.737121 22218 raft_consensus.cc:1261] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 1 FOLLOWER]: Refusing update from remote peer 3687c9265c6e4a6ea8b792a0005d39c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:49.738045 22586 consensus_queue.cc:1035] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2622434b9964e6d92005467e9a72893" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:49.738358 22590 consensus_queue.cc:1035] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fc9fff896642cfa6796e14c2bdd50a" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 35399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:49.755801 22595 mvcc.cc:205] Tried to move back new op lower bound from 7316507155392176128 to 7316507155237937152. Current Snapshot: MvccSnapshot[applied={T|T < 7316507155392176128}]
I20260809 06:23:49.771211 22055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:49.771555 22055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:49.775338 22055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a74ff86cda8413ea5cfc79af04c6374: Sending DeleteTablet for 3 replicas of tablet 178da8f230c240b0b42cee748f3a13d1
I20260809 06:23:49.776580 22486 tablet_service.cc:1514] Processing DeleteTablet for tablet 178da8f230c240b0b42cee748f3a13d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:49 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:33470
I20260809 06:23:49.777081 22195 tablet_service.cc:1514] Processing DeleteTablet for tablet 178da8f230c240b0b42cee748f3a13d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:49 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:56326
I20260809 06:23:49.778717 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 178da8f230c240b0b42cee748f3a13d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:49 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:35954
I20260809 06:23:49.781433 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22125
I20260809 06:23:49.788410 22604 tablet_replica.cc:330] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: stopping tablet replica
I20260809 06:23:49.788894 22604 raft_consensus.cc:2229] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:49.789445 22604 raft_consensus.cc:2258] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:49.790351 22605 tablet_replica.cc:330] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: stopping tablet replica
I20260809 06:23:49.790755 22605 raft_consensus.cc:2229] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:23:49.791448 22605 raft_consensus.cc:2258] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:49.792570 22603 tablet_replica.cc:330] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: stopping tablet replica
I20260809 06:23:49.793120 22603 raft_consensus.cc:2229] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:49.793666 22603 raft_consensus.cc:2258] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:49.794435 22605 ts_tablet_manager.cc:1905] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:49.795629 22603 ts_tablet_manager.cc:1905] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:49.803249 22603 ts_tablet_manager.cc:1918] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:49.803499 22603 log.cc:1196] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/178da8f230c240b0b42cee748f3a13d1
I20260809 06:23:49.803586 22605 ts_tablet_manager.cc:1918] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:49.804160 22603 ts_tablet_manager.cc:1939] T 178da8f230c240b0b42cee748f3a13d1 P b3fc9fff896642cfa6796e14c2bdd50a: Deleting consensus metadata
I20260809 06:23:49.805160 22604 ts_tablet_manager.cc:1905] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:49.806160 22043 catalog_manager.cc:4841] TS b3fc9fff896642cfa6796e14c2bdd50a (127.3.137.67:35399): tablet 178da8f230c240b0b42cee748f3a13d1 (table test-table [id=fe341efbcce0467bbd126f58c79ad8b3]) successfully deleted
I20260809 06:23:49.807154 22605 log.cc:1196] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/178da8f230c240b0b42cee748f3a13d1
I20260809 06:23:49.807863 22605 ts_tablet_manager.cc:1939] T 178da8f230c240b0b42cee748f3a13d1 P 3687c9265c6e4a6ea8b792a0005d39c3: Deleting consensus metadata
Aug 09 06:23:49 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:23:49.809657 22040 catalog_manager.cc:4841] TS 3687c9265c6e4a6ea8b792a0005d39c3 (127.3.137.66:36555): tablet 178da8f230c240b0b42cee748f3a13d1 (table test-table [id=fe341efbcce0467bbd126f58c79ad8b3]) successfully deleted
I20260809 06:23:49.813462 22604 ts_tablet_manager.cc:1918] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:49.813638 22604 log.cc:1196] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/178da8f230c240b0b42cee748f3a13d1
I20260809 06:23:49.814206 22604 ts_tablet_manager.cc:1939] T 178da8f230c240b0b42cee748f3a13d1 P e2622434b9964e6d92005467e9a72893: Deleting consensus metadata
I20260809 06:23:49.816068 22042 catalog_manager.cc:4841] TS e2622434b9964e6d92005467e9a72893 (127.3.137.65:45929): tablet 178da8f230c240b0b42cee748f3a13d1 (table test-table [id=fe341efbcce0467bbd126f58c79ad8b3]) successfully deleted
I20260809 06:23:49.818187 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.783424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.3.137.65:45929)
0809 06:23:49.783661 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.783671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.799038 (+ 15367us) server_negotiation.cc:407] Connection header received
0809 06:23:49.799434 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.799439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.799521 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.799597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:49.801477 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.802585 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.804582 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.805220 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.805330 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.810890 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:49.810916 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:49.810934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:49.810989 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:49.815184 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.816066 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.816076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.816080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.816239 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.816690 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.816693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.816695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.817085 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:49.817298 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.817667 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.817862 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:23:49.818856 22260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:49.909523 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22125
I20260809 06:23:49.938265 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22271
Aug 09 06:23:49 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:23:49.955560 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:49.939451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.3.137.66:36555)
0809 06:23:49.939670 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:49.939675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:49.939692 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:23:49.939754 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:49.939758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:49.939858 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:49.940019 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:49.941487 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.942350 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.943762 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:49.944181 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:49.944326 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:49.949203 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:49.949215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:49.949229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:49.949265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:49.952548 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.953261 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.953267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.953270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.953426 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:49.953829 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:49.953833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:49.953835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:49.954170 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:49.954530 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:49.954861 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 06:23:49.955101 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 06:23:49.956581 22407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:50.070640 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22271
I20260809 06:23:50.091971 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22420
Aug 09 06:23:50 dist-test-slave-2w3w krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:23:50.107254 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:50.092766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.3.137.67:35399)
0809 06:23:50.092955 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:50.092960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:50.093191 (+ 231us) server_negotiation.cc:407] Connection header received
0809 06:23:50.093378 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:50.093382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:50.093457 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:50.093542 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:50.094855 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:50.095640 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:50.096885 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:50.097241 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:50.097332 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:50.101955 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:50.101969 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:50.101985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:50.102015 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:50.104630 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:50.105254 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:50.105261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:50.105264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:50.105399 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:50.105729 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:50.105732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:50.105734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:50.106066 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:50.106276 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:50.106537 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 06:23:50.106691 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:23:50.107928 22554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:50.197441 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22420
I20260809 06:23:50.216367 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22022
I20260809 06:23:50.217520 22090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:50.328522 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22022
2026-08-09T06:23:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4498 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T06:23:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:23:50Z Disabled control of system clock
I20260809 06:23:50.418764 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42289
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:42289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:23:50.758056 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:50.758352 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:50.758435 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:50.765618 22615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:50.765714 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:50.765761 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:50.765846 22615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:50.765914 22615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:50.774948 22615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:42289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:50.776290 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:50.778071 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:50.790447 22621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:50.790421 22624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:50.790524 22622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:50.791499 22615 server_base.cc:1034] running on GCE node
I20260809 06:23:50.792328 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:50.793725 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:50.795009 22615 hybrid_clock.cc:648] HybridClock initialized: now 1786256630794909 us; error 42 us; skew 500 ppm
I20260809 06:23:50.795369 22615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:50.801774 22615 webserver.cc:460] Webserver started at http://127.3.137.126:42657/ using document root <none> and password file <none>
I20260809 06:23:50.802493 22615 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:50.802630 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:50.803016 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:50.805823 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "58ee80e7d2b14d5d9d2a08c763154780"
format_stamp: "Formatted at 2026-08-09 06:23:50 on dist-test-slave-2w3w"
server_key: "7c46192e9f14b7c11ae6ffc29e7243d5"
server_key_iv: "c1e0b539454548bf976041c37d42f7c6"
server_key_version: "encryptionkey@0"
I20260809 06:23:50.806633 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "58ee80e7d2b14d5d9d2a08c763154780"
format_stamp: "Formatted at 2026-08-09 06:23:50 on dist-test-slave-2w3w"
server_key: "7c46192e9f14b7c11ae6ffc29e7243d5"
server_key_iv: "c1e0b539454548bf976041c37d42f7c6"
server_key_version: "encryptionkey@0"
I20260809 06:23:50.812078 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 06:23:50.815462 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:50.816810 22615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:23:50.817008 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "58ee80e7d2b14d5d9d2a08c763154780"
format_stamp: "Formatted at 2026-08-09 06:23:50 on dist-test-slave-2w3w"
server_key: "7c46192e9f14b7c11ae6ffc29e7243d5"
server_key_iv: "c1e0b539454548bf976041c37d42f7c6"
server_key_version: "encryptionkey@0"
I20260809 06:23:50.817222 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:50.828866 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:50.829658 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:50.829975 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:50.849116 22615 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:42289
I20260809 06:23:50.849135 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:42289 every 8 connection(s)
I20260809 06:23:50.850407 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:50.853587 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22615
I20260809 06:23:50.853755 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:23:50.854352 3621 external_mini_cluster.cc:1428] Setting key 566c3304b53e9deb30ccd5e8b45869ff
I20260809 06:23:50.854378 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:50.867712 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:50.856272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.3.137.126:42289)
0809 06:23:50.857227 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:50.857239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:50.857281 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:23:50.858875 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:50.858895 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:50.859377 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:50.859686 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:50.860307 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:50.861593 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:50.863469 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:50.864110 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:50.864247 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:50.865474 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:50.865484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:50.865497 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:50.866040 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:50.866117 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:50.866136 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:50.866222 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:50.866453 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 06:23:50.866759 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":363,"threads_started":1}
I20260809 06:23:50.867879 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: Bootstrap starting.
I20260809 06:23:50.870820 22690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:50.871793 22690 log.cc:824] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:50.874393 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: No bootstrap required, opened a new log
I20260809 06:23:50.881290 22690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } }
I20260809 06:23:50.881644 22690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:50.881788 22690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ee80e7d2b14d5d9d2a08c763154780, State: Initialized, Role: FOLLOWER
I20260809 06:23:50.882318 22690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } }
I20260809 06:23:50.882567 22690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:50.882712 22690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:50.882889 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:50.885015 22690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } }
I20260809 06:23:50.885499 22690 leader_election.cc:302] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58ee80e7d2b14d5d9d2a08c763154780; no voters:
I20260809 06:23:50.886149 22690 leader_election.cc:288] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:23:50.886315 22695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:50.887446 22695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [term 1 LEADER]: Becoming Leader. State: Replica: 58ee80e7d2b14d5d9d2a08c763154780, State: Running, Role: LEADER
I20260809 06:23:50.887964 22695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } }
I20260809 06:23:50.888425 22690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:50.891043 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58ee80e7d2b14d5d9d2a08c763154780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } } }
I20260809 06:23:50.891067 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58ee80e7d2b14d5d9d2a08c763154780. Latest consensus state: current_term: 1 leader_uuid: "58ee80e7d2b14d5d9d2a08c763154780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ee80e7d2b14d5d9d2a08c763154780" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42289 } } }
I20260809 06:23:50.891522 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:50.891592 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780 [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:50.893772 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:50.898896 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:50.907155 22704 catalog_manager.cc:1347] Generated new cluster ID: 817ef3c57db64c0f864ad2c9e6072eeb
I20260809 06:23:50.907261 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:50.918519 22704 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:23:50.919806 22704 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:50.936786 22704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: Generated new TSK 0
I20260809 06:23:50.937579 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:23:50.948333 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--builtin_ntp_servers=127.3.137.84:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:23:51.297040 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:51.297343 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:51.297493 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:51.305547 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:51.305648 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:51.305846 22715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:51.315642 22715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:51.316936 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:51.318775 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:51.332614 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:51.332705 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:51.332736 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:51.333637 22715 server_base.cc:1034] running on GCE node
I20260809 06:23:51.334211 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:51.335235 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:51.336478 22715 hybrid_clock.cc:648] HybridClock initialized: now 1786256631336434 us; error 36 us; skew 500 ppm
I20260809 06:23:51.336825 22715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:51.339316 22715 webserver.cc:460] Webserver started at http://127.3.137.65:41689/ using document root <none> and password file <none>
I20260809 06:23:51.340058 22715 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:51.340204 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:51.340569 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:51.343797 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "a81f207151a74ddd925adb818145c4a9"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "040444b6f796e2f9f4cf2b365ebcb85c"
server_key_iv: "e801d74ed7a0905bc6671064fcbc4550"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.344646 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "a81f207151a74ddd925adb818145c4a9"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "040444b6f796e2f9f4cf2b365ebcb85c"
server_key_iv: "e801d74ed7a0905bc6671064fcbc4550"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.349918 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:23:51.353653 22729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:51.355039 22715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:51.355214 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "a81f207151a74ddd925adb818145c4a9"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "040444b6f796e2f9f4cf2b365ebcb85c"
server_key_iv: "e801d74ed7a0905bc6671064fcbc4550"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.355443 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:51.387909 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:51.388772 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:51.389076 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:51.390127 22715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:51.391615 22715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:51.391747 22715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:51.391873 22715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:51.391963 22715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:51.424489 22715 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:37231
I20260809 06:23:51.424536 22852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:37231 every 8 connection(s)
I20260809 06:23:51.425781 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:51.429486 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22715
I20260809 06:23:51.429665 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:23:51.430137 3621 external_mini_cluster.cc:1428] Setting key 2e2e6e9cddbcc8d3dee5011c74969276
I20260809 06:23:51.433030 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--builtin_ntp_servers=127.3.137.84:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:23:51.439667 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:51.428378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:54917 (local address 127.3.137.126:42289)
0809 06:23:51.428883 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:51.428890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:51.430223 (+ 1333us) server_negotiation.cc:407] Connection header received
0809 06:23:51.432214 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:51.432219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:51.432312 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:51.432429 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:51.433565 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.434764 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.436428 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.437167 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.437310 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:51.438598 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:51.438607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:51.438614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:51.438640 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:51.438696 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:51.438714 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:51.438809 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:51.439100 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 06:23:51.439215 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":206,"threads_started":1}
I20260809 06:23:51.440383 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:51.429031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42289 (local address 127.3.137.65:54917)
0809 06:23:51.430053 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:51.430097 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:51.431965 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:51.432625 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:51.432633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:51.432717 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:51.433389 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:51.433408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.434926 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.434930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:51.436247 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:51.436262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.437372 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.437377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:51.437585 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:51.438291 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:51.438293 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:51.438329 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:51.438401 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:51.438832 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:51.438847 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:51.438858 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 06:23:51.439062 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 06:23:51.439344 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":332,"threads_started":1}
I20260809 06:23:51.441900 22853 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42289
I20260809 06:23:51.442231 22853 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:51.443019 22853 heartbeater.cc:511] Master 127.3.137.126:42289 requested a full tablet report, sending...
I20260809 06:23:51.445060 22649 ts_manager.cc:194] Registered new tserver with Master: a81f207151a74ddd925adb818145c4a9 (127.3.137.65:37231)
I20260809 06:23:51.447104 22649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.65:54917
W20260809 06:23:51.766674 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:51.766939 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:51.767107 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:51.774358 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:51.774453 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:51.774621 22857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:51.783541 22857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:51.784806 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:51.786625 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:51.800448 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:51.800635 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:51.800635 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:51.801842 22857 server_base.cc:1034] running on GCE node
I20260809 06:23:51.802384 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:51.803350 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:51.804555 22857 hybrid_clock.cc:648] HybridClock initialized: now 1786256631804508 us; error 38 us; skew 500 ppm
I20260809 06:23:51.804824 22857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:51.807072 22857 webserver.cc:460] Webserver started at http://127.3.137.66:35501/ using document root <none> and password file <none>
I20260809 06:23:51.807703 22857 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:51.807789 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:51.808074 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:51.810768 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "5dbbddff7bdf4fae8e830f4bd2958029"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "841b2d46dd3b3de6372fd69f877ba64a"
server_key_iv: "e88252e599da28dde4508438bbf1cd0d"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.811689 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "5dbbddff7bdf4fae8e830f4bd2958029"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "841b2d46dd3b3de6372fd69f877ba64a"
server_key_iv: "e88252e599da28dde4508438bbf1cd0d"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.816746 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:23:51.820178 22873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:51.821534 22857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:23:51.821700 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "5dbbddff7bdf4fae8e830f4bd2958029"
format_stamp: "Formatted at 2026-08-09 06:23:51 on dist-test-slave-2w3w"
server_key: "841b2d46dd3b3de6372fd69f877ba64a"
server_key_iv: "e88252e599da28dde4508438bbf1cd0d"
server_key_version: "encryptionkey@0"
I20260809 06:23:51.821900 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:51.835245 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:51.836019 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:51.836314 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:51.837237 22857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:51.838615 22857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:51.838721 22857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:51.838894 22857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:51.839064 22857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 06:23:51.876938 22857 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:34431
I20260809 06:23:51.876992 22996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:34431 every 8 connection(s)
I20260809 06:23:51.878243 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:51.884801 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 22857
I20260809 06:23:51.885002 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:23:51.885594 3621 external_mini_cluster.cc:1428] Setting key ae31076cf71117cc1d05fcb5ad518c60
I20260809 06:23:51.888536 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--builtin_ntp_servers=127.3.137.84:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:23:51.891376 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:51.880832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:46351 (local address 127.3.137.126:42289)
0809 06:23:51.881053 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:51.881060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:51.882483 (+ 1423us) server_negotiation.cc:407] Connection header received
0809 06:23:51.884302 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:51.884307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:51.884376 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:51.884457 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:51.885889 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.886744 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.888431 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.888996 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.889148 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:51.890557 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:51.890561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:51.890565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:51.890582 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:51.890611 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:51.890623 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:51.890716 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:51.890945 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 06:23:51.891067 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:23:51.892472 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:51.881337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42289 (local address 127.3.137.66:46351)
0809 06:23:51.882334 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:51.882375 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:51.884043 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:51.884746 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:51.884758 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:51.884875 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:51.885694 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:51.885721 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.886911 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.886917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:51.888282 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:51.888295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:51.889234 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:51.889242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:51.889531 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:51.890243 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:51.890246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:51.890282 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:51.890363 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:51.890733 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:51.890744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:51.890751 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:23:51.890932 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 06:23:51.891350 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":431,"threads_started":1}
I20260809 06:23:51.893874 22998 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42289
I20260809 06:23:51.894187 22998 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:51.895156 22998 heartbeater.cc:511] Master 127.3.137.126:42289 requested a full tablet report, sending...
I20260809 06:23:51.896710 22649 ts_manager.cc:194] Registered new tserver with Master: 5dbbddff7bdf4fae8e830f4bd2958029 (127.3.137.66:34431)
I20260809 06:23:51.897883 22649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.66:46351
W20260809 06:23:52.218070 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:52.218416 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:52.218592 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:52.226116 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:52.226243 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:52.226413 23001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:52.235495 23001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:52.236790 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:52.238518 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:52.251705 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:52.251657 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:52.251600 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:52.252569 23001 server_base.cc:1034] running on GCE node
I20260809 06:23:52.253158 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:52.254072 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:52.255307 23001 hybrid_clock.cc:648] HybridClock initialized: now 1786256632255264 us; error 47 us; skew 500 ppm
I20260809 06:23:52.255625 23001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:52.257921 23001 webserver.cc:460] Webserver started at http://127.3.137.67:44973/ using document root <none> and password file <none>
I20260809 06:23:52.258590 23001 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:52.258726 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:52.259148 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:52.262120 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "d2d99053906e42408f4055632f8444b8"
format_stamp: "Formatted at 2026-08-09 06:23:52 on dist-test-slave-2w3w"
server_key: "b82cce3dd327b44dad7a31c91152e970"
server_key_iv: "4d528f79a758b0b291f0c0a1e8cb8d7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:52.262928 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "d2d99053906e42408f4055632f8444b8"
format_stamp: "Formatted at 2026-08-09 06:23:52 on dist-test-slave-2w3w"
server_key: "b82cce3dd327b44dad7a31c91152e970"
server_key_iv: "4d528f79a758b0b291f0c0a1e8cb8d7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:52.268023 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:23:52.271267 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:52.272497 23001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260809 06:23:52.272670 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "d2d99053906e42408f4055632f8444b8"
format_stamp: "Formatted at 2026-08-09 06:23:52 on dist-test-slave-2w3w"
server_key: "b82cce3dd327b44dad7a31c91152e970"
server_key_iv: "4d528f79a758b0b291f0c0a1e8cb8d7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:52.272894 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:52.283445 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:52.284173 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:52.284474 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:52.285385 23001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:52.286763 23001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:52.286895 23001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:52.287041 23001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:52.287142 23001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:52.321408 23001 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:32905
I20260809 06:23:52.321501 23138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:32905 every 8 connection(s)
I20260809 06:23:52.322721 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:52.329134 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23001
I20260809 06:23:52.329321 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:23:52.329785 3621 external_mini_cluster.cc:1428] Setting key 9206e417f90d9e6787501be33b78c35a
I20260809 06:23:52.335281 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.325370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:33389 (local address 127.3.137.126:42289)
0809 06:23:52.325552 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.325557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.327069 (+ 1512us) server_negotiation.cc:407] Connection header received
0809 06:23:52.328818 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.328825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.328917 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.329040 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:52.330176 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.331033 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.332782 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.333195 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.333289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.334498 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:52.334503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:52.334506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:52.334524 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:52.334556 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:52.334568 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:52.334635 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.334901 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.335054 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:23:52.336267 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.325917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42289 (local address 127.3.137.67:33389)
0809 06:23:52.326888 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.326928 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.328542 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.329235 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.329248 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.329390 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:52.330009 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.330028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.331180 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.331185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.332657 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.332670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.333385 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.333390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.333596 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.334183 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:52.334186 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:52.334247 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:52.334324 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:52.334668 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:52.334678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:52.334685 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.334894 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.335213 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":326,"threads_started":1}
I20260809 06:23:52.337572 23139 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42289
I20260809 06:23:52.337896 23139 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:52.338608 23139 heartbeater.cc:511] Master 127.3.137.126:42289 requested a full tablet report, sending...
I20260809 06:23:52.340020 22649 ts_manager.cc:194] Registered new tserver with Master: d2d99053906e42408f4055632f8444b8 (127.3.137.67:32905)
I20260809 06:23:52.340840 22649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.67:33389
I20260809 06:23:52.344674 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:23:52.354292 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.348641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.3.137.126:42289)
0809 06:23:52.348802 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.348806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.349369 (+ 563us) server_negotiation.cc:407] Connection header received
0809 06:23:52.349499 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.349502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.349558 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.349637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:52.350142 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.350814 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.352269 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.352700 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.352791 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.353749 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:52.353754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:52.353757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:52.353773 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:52.353804 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:52.353815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:52.353884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.354001 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.354069 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:23:52.357983 22649 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:23:52.364768 22649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:23:52.389312 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.377098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:35982)
0809 06:23:52.380010 (+ 2912us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.380029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.380165 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.380887 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.380893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.380924 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:52.381374 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.381390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.383299 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.383304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.385866 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.385881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.388000 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.388010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.388391 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.388543 (+ 152us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.388735 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.388897 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2506,"mutex_wait_us":284,"thread_start_us":2357,"threads_started":1}
I20260809 06:23:52.389394 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.377729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.3.137.65:37231)
0809 06:23:52.379309 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.379320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.380310 (+ 990us) server_negotiation.cc:407] Connection header received
0809 06:23:52.380435 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.380442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.380672 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.380886 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:52.381559 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.383148 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.386080 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.387465 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.387603 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.387847 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.388811 (+ 964us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.388984 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":1299,"threads_started":1}
I20260809 06:23:52.390065 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.377056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.3.137.67:32905)
0809 06:23:52.378010 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.378022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.378610 (+ 588us) server_negotiation.cc:407] Connection header received
0809 06:23:52.378924 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.378936 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.379203 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.380154 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:52.381295 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.383079 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.385925 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.387465 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.387603 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.387882 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.389091 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.389279 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":543,"threads_started":1}
I20260809 06:23:52.390079 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.376639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:32905 (local address 127.0.0.1:52784)
0809 06:23:52.378466 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.378502 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.378687 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.380550 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.380557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.380587 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:52.381032 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.381054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.383224 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.383231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.385694 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.385707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.388054 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.388059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.388393 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.388543 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.388673 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.388891 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":265,"thread_start_us":1391,"threads_started":1}
I20260809 06:23:52.390339 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.380420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.3.137.66:34431)
0809 06:23:52.381547 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.381557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.382116 (+ 559us) server_negotiation.cc:407] Connection header received
0809 06:23:52.382299 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.382305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.382460 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.382656 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:52.383336 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.384863 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.386908 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.388352 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.388495 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.389754 (+ 1259us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.389838 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.389994 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":835,"threads_started":1}
I20260809 06:23:52.390652 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.379504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:34431 (local address 127.0.0.1:57770)
0809 06:23:52.381972 (+ 2468us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.381987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.382142 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.382658 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.382662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.382688 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:52.383097 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.383107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.385012 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.385018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.386706 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.386718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.388739 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.388747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.388932 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.388963 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.389039 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.389187 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2376,"thread_start_us":2249,"threads_started":1}
I20260809 06:23:52.393527 22782 tablet_service.cc:1467] Processing CreateTablet for tablet 81d2405b1250495c8a76eced7e986840 (DEFAULT_TABLE table=test-table [id=2fdafb09b5f347ac92bab86c5a488b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:52.394102 22927 tablet_service.cc:1467] Processing CreateTablet for tablet 81d2405b1250495c8a76eced7e986840 (DEFAULT_TABLE table=test-table [id=2fdafb09b5f347ac92bab86c5a488b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:52.395485 22782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d2405b1250495c8a76eced7e986840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:52.395507 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d2405b1250495c8a76eced7e986840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:52.395341 23070 tablet_service.cc:1467] Processing CreateTablet for tablet 81d2405b1250495c8a76eced7e986840 (DEFAULT_TABLE table=test-table [id=2fdafb09b5f347ac92bab86c5a488b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:52.396682 23070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d2405b1250495c8a76eced7e986840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:52.407909 23158 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Bootstrap starting.
I20260809 06:23:52.410063 23160 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Bootstrap starting.
I20260809 06:23:52.412261 23158 tablet_bootstrap.cc:654] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:52.413295 23160 tablet_bootstrap.cc:654] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:52.413606 23158 log.cc:824] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:52.414499 23160 log.cc:824] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:52.415684 23159 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: Bootstrap starting.
I20260809 06:23:52.418529 23158 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: No bootstrap required, opened a new log
I20260809 06:23:52.418536 23160 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: No bootstrap required, opened a new log
I20260809 06:23:52.418795 23160 ts_tablet_manager.cc:1397] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 06:23:52.418814 23158 ts_tablet_manager.cc:1397] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260809 06:23:52.419153 23159 tablet_bootstrap.cc:654] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:52.420375 23159 log.cc:824] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:52.423560 23159 tablet_bootstrap.cc:492] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: No bootstrap required, opened a new log
I20260809 06:23:52.423839 23159 ts_tablet_manager.cc:1397] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260809 06:23:52.428009 23160 raft_consensus.cc:348] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.428429 23160 raft_consensus.cc:374] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:52.428534 23160 raft_consensus.cc:729] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbbddff7bdf4fae8e830f4bd2958029, State: Initialized, Role: FOLLOWER
I20260809 06:23:52.429127 23160 consensus_queue.cc:260] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.430225 22998 heartbeater.cc:503] Master 127.3.137.126:42289 was elected leader, sending a full tablet report...
I20260809 06:23:52.430027 23158 raft_consensus.cc:348] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.430464 23158 raft_consensus.cc:374] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:52.430553 23158 raft_consensus.cc:729] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d99053906e42408f4055632f8444b8, State: Initialized, Role: FOLLOWER
I20260809 06:23:52.430927 23160 ts_tablet_manager.cc:1428] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 06:23:52.431123 23158 consensus_queue.cc:260] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.438668 23159 raft_consensus.cc:348] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.439091 23159 raft_consensus.cc:374] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:52.439219 23159 raft_consensus.cc:729] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81f207151a74ddd925adb818145c4a9, State: Initialized, Role: FOLLOWER
I20260809 06:23:52.439874 23159 consensus_queue.cc:260] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.440348 23158 ts_tablet_manager.cc:1428] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20260809 06:23:52.441469 23139 heartbeater.cc:503] Master 127.3.137.126:42289 was elected leader, sending a full tablet report...
I20260809 06:23:52.441676 22853 heartbeater.cc:503] Master 127.3.137.126:42289 was elected leader, sending a full tablet report...
I20260809 06:23:52.442325 23159 ts_tablet_manager.cc:1428] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: Time spent starting tablet: real 0.018s user 0.006s sys 0.007s
I20260809 06:23:52.494462 23166 raft_consensus.cc:482] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:52.494760 23166 raft_consensus.cc:504] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.496531 23166 leader_election.cc:288] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d99053906e42408f4055632f8444b8 (127.3.137.67:32905), 5dbbddff7bdf4fae8e830f4bd2958029 (127.3.137.66:34431)
I20260809 06:23:52.503978 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.497014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:34115 (local address 127.3.137.67:32905)
0809 06:23:52.497274 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.497284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.497776 (+ 492us) server_negotiation.cc:407] Connection header received
0809 06:23:52.498046 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.498054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.498138 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.498219 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:52.499108 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.500001 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.502180 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.503060 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.503185 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.503292 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.503587 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.503725 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:23:52.504026 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.498076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:42423 (local address 127.3.137.66:34431)
0809 06:23:52.498286 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.498292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.498448 (+ 156us) server_negotiation.cc:407] Connection header received
0809 06:23:52.498562 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.498568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.498649 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.498760 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:52.499369 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.500247 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.502295 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.503138 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.503308 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.503376 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.503672 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.503800 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:23:52.504081 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.496909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:32905 (local address 127.3.137.65:34115)
0809 06:23:52.497606 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.497654 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.497832 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.498465 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.498470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.498506 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:52.498903 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.498927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.500150 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.500156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.501974 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.501986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.503251 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.503257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.503435 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.503461 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.503532 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.503667 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":412,"threads_started":1}
I20260809 06:23:52.504103 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.497679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:34431 (local address 127.3.137.65:42423)
0809 06:23:52.498338 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.498353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:52.498457 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:52.498771 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:52.498774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:52.498792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:52.499191 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.499199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.500420 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.500425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.502096 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:52.502116 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.503355 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.503362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:52.503488 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.503532 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 06:23:52.503626 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 06:23:52.503740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"thread_start_us":365,"threads_started":1}
I20260809 06:23:52.505218 23092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d2405b1250495c8a76eced7e986840" candidate_uuid: "a81f207151a74ddd925adb818145c4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d99053906e42408f4055632f8444b8" is_pre_election: true
I20260809 06:23:52.505200 22948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d2405b1250495c8a76eced7e986840" candidate_uuid: "a81f207151a74ddd925adb818145c4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" is_pre_election: true
I20260809 06:23:52.505798 22948 raft_consensus.cc:2454] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a81f207151a74ddd925adb818145c4a9 in term 0.
I20260809 06:23:52.505798 23092 raft_consensus.cc:2454] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a81f207151a74ddd925adb818145c4a9 in term 0.
I20260809 06:23:52.506531 22733 leader_election.cc:302] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a81f207151a74ddd925adb818145c4a9, d2d99053906e42408f4055632f8444b8; no voters:
I20260809 06:23:52.506908 23166 raft_consensus.cc:2781] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:23:52.507092 23166 raft_consensus.cc:482] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:23:52.507220 23166 raft_consensus.cc:3037] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:52.509383 23166 raft_consensus.cc:504] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.509975 23166 leader_election.cc:288] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [CANDIDATE]: Term 1 election: Requested vote from peers d2d99053906e42408f4055632f8444b8 (127.3.137.67:32905), 5dbbddff7bdf4fae8e830f4bd2958029 (127.3.137.66:34431)
I20260809 06:23:52.510450 22948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d2405b1250495c8a76eced7e986840" candidate_uuid: "a81f207151a74ddd925adb818145c4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dbbddff7bdf4fae8e830f4bd2958029"
I20260809 06:23:52.510458 23092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d2405b1250495c8a76eced7e986840" candidate_uuid: "a81f207151a74ddd925adb818145c4a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d99053906e42408f4055632f8444b8"
I20260809 06:23:52.510653 22948 raft_consensus.cc:3037] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:52.510668 23092 raft_consensus.cc:3037] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:52.512923 22948 raft_consensus.cc:2454] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a81f207151a74ddd925adb818145c4a9 in term 1.
I20260809 06:23:52.512991 23092 raft_consensus.cc:2454] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a81f207151a74ddd925adb818145c4a9 in term 1.
I20260809 06:23:52.513366 22733 leader_election.cc:302] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dbbddff7bdf4fae8e830f4bd2958029, a81f207151a74ddd925adb818145c4a9; no voters:
I20260809 06:23:52.513669 23166 raft_consensus.cc:2781] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:52.514103 23166 raft_consensus.cc:686] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 1 LEADER]: Becoming Leader. State: Replica: a81f207151a74ddd925adb818145c4a9, State: Running, Role: LEADER
I20260809 06:23:52.514559 23166 consensus_queue.cc:237] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } }
I20260809 06:23:52.519361 22648 catalog_manager.cc:5495] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a81f207151a74ddd925adb818145c4a9 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "a81f207151a74ddd925adb818145c4a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a81f207151a74ddd925adb818145c4a9" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 37231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 } health_report { overall_health: UNKNOWN } } }
I20260809 06:23:52.541683 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.536708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.3.137.65:37231)
0809 06:23:52.536895 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.536900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.536916 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:23:52.537053 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.537057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.537140 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.537256 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:52.537576 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.538282 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.539679 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.540094 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.540186 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.540613 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:52.541118 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:23:52.541231 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.541339 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.541400 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:23:52.549774 22948 raft_consensus.cc:1261] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 1 FOLLOWER]: Refusing update from remote peer a81f207151a74ddd925adb818145c4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:52.549765 23092 raft_consensus.cc:1261] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 1 FOLLOWER]: Refusing update from remote peer a81f207151a74ddd925adb818145c4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:52.550714 23169 consensus_queue.cc:1035] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d99053906e42408f4055632f8444b8" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:52.551157 23166 consensus_queue.cc:1035] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dbbddff7bdf4fae8e830f4bd2958029" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:52.561844 23172 mvcc.cc:205] Tried to move back new op lower bound from 7316507166914600960 to 7316507166783463424. Current Snapshot: MvccSnapshot[applied={T|T < 7316507166914600960}]
I20260809 06:23:52.575795 22648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:52.576370 22648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:52.580226 22648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58ee80e7d2b14d5d9d2a08c763154780: Sending DeleteTablet for 3 replicas of tablet 81d2405b1250495c8a76eced7e986840
I20260809 06:23:52.581357 22782 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d2405b1250495c8a76eced7e986840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:52 UTC) from {username='slave'} at 127.0.0.1:35982
I20260809 06:23:52.581432 23070 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d2405b1250495c8a76eced7e986840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:52 UTC) from {username='slave'} at 127.0.0.1:52784
I20260809 06:23:52.582432 22927 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d2405b1250495c8a76eced7e986840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:52 UTC) from {username='slave'} at 127.0.0.1:57770
I20260809 06:23:52.582576 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22715
I20260809 06:23:52.596205 23183 tablet_replica.cc:330] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: stopping tablet replica
I20260809 06:23:52.596819 23183 raft_consensus.cc:2229] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:52.597500 23183 raft_consensus.cc:2258] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:52.598448 23184 tablet_replica.cc:330] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: stopping tablet replica
I20260809 06:23:52.598976 23184 raft_consensus.cc:2229] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:52.599572 23184 raft_consensus.cc:2258] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:52.602665 23183 ts_tablet_manager.cc:1905] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:52.608152 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.583871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.3.137.65:37231)
0809 06:23:52.586177 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.586184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.598721 (+ 12537us) server_negotiation.cc:407] Connection header received
0809 06:23:52.599078 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.599085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.599193 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.599278 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:52.600311 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.601567 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.603708 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.604434 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.604539 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.606121 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:52.606132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:52.606149 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:52.607148 (+ 999us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:52.607215 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:52.607240 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:52.607329 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.607615 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.607737 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165}
I20260809 06:23:52.609261 22848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:52.608494 23182 tablet_replica.cc:330] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9: stopping tablet replica
I20260809 06:23:52.610726 23184 ts_tablet_manager.cc:1905] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:52.614593 23183 ts_tablet_manager.cc:1918] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:52.614779 23183 log.cc:1196] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/81d2405b1250495c8a76eced7e986840
I20260809 06:23:52.615340 23183 ts_tablet_manager.cc:1939] T 81d2405b1250495c8a76eced7e986840 P d2d99053906e42408f4055632f8444b8: Deleting consensus metadata
I20260809 06:23:52.616791 23184 ts_tablet_manager.cc:1918] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:52.616961 23184 log.cc:1196] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/81d2405b1250495c8a76eced7e986840
I20260809 06:23:52.617234 22634 catalog_manager.cc:4841] TS d2d99053906e42408f4055632f8444b8 (127.3.137.67:32905): tablet 81d2405b1250495c8a76eced7e986840 (table test-table [id=2fdafb09b5f347ac92bab86c5a488b3e]) successfully deleted
I20260809 06:23:52.617581 23184 ts_tablet_manager.cc:1939] T 81d2405b1250495c8a76eced7e986840 P 5dbbddff7bdf4fae8e830f4bd2958029: Deleting consensus metadata
I20260809 06:23:52.618927 22634 catalog_manager.cc:4841] TS 5dbbddff7bdf4fae8e830f4bd2958029 (127.3.137.66:34431): tablet 81d2405b1250495c8a76eced7e986840 (table test-table [id=2fdafb09b5f347ac92bab86c5a488b3e]) successfully deleted
I20260809 06:23:52.610621 23182 raft_consensus.cc:2229] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:23:52.708894 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22715
I20260809 06:23:52.708902 23182 raft_consensus.cc:2258] T 81d2405b1250495c8a76eced7e986840 P a81f207151a74ddd925adb818145c4a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:52.728935 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22857
W20260809 06:23:52.728946 22632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:37231 (error 108)
I20260809 06:23:52.731108 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.730583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36016)
0809 06:23:52.730787 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.730892 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260809 06:23:52.731362 22632 catalog_manager.cc:4584] TS a81f207151a74ddd925adb818145c4a9 (127.3.137.65:37231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81d2405b1250495c8a76eced7e986840: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
I20260809 06:23:52.735944 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.729783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.3.137.66:34431)
0809 06:23:52.729998 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.730005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.730131 (+ 126us) server_negotiation.cc:407] Connection header received
0809 06:23:52.730295 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.730301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.730428 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.730520 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:52.731052 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.731912 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.733197 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.733579 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.733670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.734619 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:52.734628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:52.734640 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:52.735327 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:52.735375 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:52.735395 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:52.735490 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.735609 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.735694 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:23:52.736624 22991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:52.785148 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.784568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36022)
0809 06:23:52.784872 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.784970 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260809 06:23:52.786202 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.785884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36028)
0809 06:23:52.786017 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.786087 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 06:23:52.828198 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22857
I20260809 06:23:52.847014 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.846356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36040)
0809 06:23:52.846707 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.846782 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260809 06:23:52.847014 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23001
I20260809 06:23:52.848174 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.847786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36044)
0809 06:23:52.847942 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.848032 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 06:23:52.854732 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.847863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.3.137.67:32905)
0809 06:23:52.848056 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:52.848062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:52.848083 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:23:52.848210 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:52.848219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:52.848342 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:52.848489 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:52.848915 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.850062 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.851439 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:52.852085 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:52.852204 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:52.853253 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:52.853267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:52.853289 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:52.854021 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:52.854085 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:52.854112 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:52.854236 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:52.854314 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 06:23:52.854415 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:23:52.855391 23133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:52.928781 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.928312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36058)
0809 06:23:52.928537 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.928624 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:23:52.929935 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:52.929542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:37231 (local address 127.0.0.1:36074)
0809 06:23:52.929749 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:52.929825 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 06:23:52.945181 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23001
I20260809 06:23:52.963617 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 22615
I20260809 06:23:52.964879 22683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:53.080163 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 22615
2026-08-09T06:23:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2747 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:23:53 dist-test-slave-2w3w krb5kdc[23195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:23:53 dist-test-slave-2w3w krb5kdc[23195](info): set up 2 sockets
Aug 09 06:23:53 dist-test-slave-2w3w krb5kdc[23195](info): commencing operation
krb5kdc: starting...
W20260809 06:23:54.244177 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:23:54 dist-test-slave-2w3w krb5kdc[23195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:23:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:23:54.544555 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:36299
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:54.884434 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:54.884711 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:54.884923 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:54.892268 23209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:54.892383 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:54.892493 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:54.892594 23209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:54.892678 23209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:54.902822 23209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:36299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:54.904198 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:54.905468 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:54.917351 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:54.917428 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:54.917537 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:54.918234 23209 server_base.cc:1034] running on GCE node
I20260809 06:23:54.919088 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:54.920469 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:54.921715 23209 hybrid_clock.cc:648] HybridClock initialized: now 1786256634921670 us; error 52 us; skew 500 ppm
I20260809 06:23:54.922106 23209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:54 dist-test-slave-2w3w krb5kdc[23195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:54.925336 23209 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:23:54.926854 23209 webserver.cc:460] Webserver started at http://127.3.137.126:33491/ using document root <none> and password file <none>
I20260809 06:23:54.927589 23209 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:54.927742 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:54.928087 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:54.931173 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "71208be4afad4fb8b40374470440081b"
format_stamp: "Formatted at 2026-08-09 06:23:54 on dist-test-slave-2w3w"
server_key: "915791c568afa4ac59065c5f1570c205"
server_key_iv: "f926fc1819d2b91fe8cf4158b9dd89e1"
server_key_version: "encryptionkey@0"
I20260809 06:23:54.931953 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "71208be4afad4fb8b40374470440081b"
format_stamp: "Formatted at 2026-08-09 06:23:54 on dist-test-slave-2w3w"
server_key: "915791c568afa4ac59065c5f1570c205"
server_key_iv: "f926fc1819d2b91fe8cf4158b9dd89e1"
server_key_version: "encryptionkey@0"
I20260809 06:23:54.937109 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:23:54.940568 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:54.941808 23209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:23:54.942008 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "71208be4afad4fb8b40374470440081b"
format_stamp: "Formatted at 2026-08-09 06:23:54 on dist-test-slave-2w3w"
server_key: "915791c568afa4ac59065c5f1570c205"
server_key_iv: "f926fc1819d2b91fe8cf4158b9dd89e1"
server_key_version: "encryptionkey@0"
I20260809 06:23:54.942212 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:54.957442 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:54.961505 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:54.961793 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:54.980931 23209 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:36299
I20260809 06:23:54.980952 23280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:36299 every 8 connection(s)
I20260809 06:23:54.982419 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:54.983930 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23209
I20260809 06:23:54.984143 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:23:54.984697 3621 external_mini_cluster.cc:1428] Setting key bb7dbbef42858e86732c76753f5ae82f
I20260809 06:23:54.987267 23281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:54.997727 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: Bootstrap starting.
Aug 09 06:23:54 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:55.001368 23281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:55.002425 23281 log.cc:824] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:55.005874 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: No bootstrap required, opened a new log
I20260809 06:23:55.007977 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:54.986651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.3.137.126:36299)
0809 06:23:54.987527 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:54.987539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:54.987585 (+ 46us) server_negotiation.cc:407] Connection header received
0809 06:23:54.989224 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:54.989245 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:54.989610 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:54.990042 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:54.991691 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:54.993013 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:54.994408 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:54.995202 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:54.995431 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:55.000021 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:55.000033 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:55.000050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:55.000093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:55.003725 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:55.004646 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:55.004653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:55.004659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:55.004862 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:55.005329 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:55.005332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:55.005335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:55.005853 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:55.006141 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:55.006579 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 06:23:55.006965 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":351,"threads_started":1}
I20260809 06:23:55.014565 23281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } }
I20260809 06:23:55.014932 23281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:55.015056 23281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71208be4afad4fb8b40374470440081b, State: Initialized, Role: FOLLOWER
I20260809 06:23:55.015614 23281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } }
I20260809 06:23:55.015882 23281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:55.016041 23281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:55.016242 23281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:55.018421 23281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } }
I20260809 06:23:55.018920 23281 leader_election.cc:302] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71208be4afad4fb8b40374470440081b; no voters:
I20260809 06:23:55.019572 23281 leader_election.cc:288] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:23:55.019718 23286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:55.020965 23286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [term 1 LEADER]: Becoming Leader. State: Replica: 71208be4afad4fb8b40374470440081b, State: Running, Role: LEADER
I20260809 06:23:55.021456 23286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } }
I20260809 06:23:55.021624 23281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:55.024753 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71208be4afad4fb8b40374470440081b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } } }
I20260809 06:23:55.024748 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71208be4afad4fb8b40374470440081b. Latest consensus state: current_term: 1 leader_uuid: "71208be4afad4fb8b40374470440081b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71208be4afad4fb8b40374470440081b" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36299 } } }
I20260809 06:23:55.025148 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:55.025210 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:55.027429 23295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:55.031926 23295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:55.040307 23295 catalog_manager.cc:1347] Generated new cluster ID: e219b52753804ff8981f314a8cc6f856
I20260809 06:23:55.040417 23295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:55.052033 23295 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:23:55.053290 23295 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:55.063485 23295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: Generated new TSK 0
I20260809 06:23:55.064236 23295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:23:55.132539 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:55.468608 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:55.468880 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:55.468971 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:55.476118 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:55.476233 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:55.476415 23310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:55.486037 23310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:55.487437 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:55.488795 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:55.502759 23318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:55.502846 23316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:55.502848 23315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:55.503566 23310 server_base.cc:1034] running on GCE node
I20260809 06:23:55.504127 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:55.504981 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:55.506174 23310 hybrid_clock.cc:648] HybridClock initialized: now 1786256635506114 us; error 51 us; skew 500 ppm
I20260809 06:23:55.506489 23310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:55 dist-test-slave-2w3w krb5kdc[23195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:55.509732 23310 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:23:55.511242 23310 webserver.cc:460] Webserver started at http://127.3.137.65:44217/ using document root <none> and password file <none>
I20260809 06:23:55.511900 23310 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:55.512053 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:55.512380 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:55.515265 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "4a3d7c9b40834939b451c7a2ba1ac974"
format_stamp: "Formatted at 2026-08-09 06:23:55 on dist-test-slave-2w3w"
server_key: "06318bc5af7d86e8d36c0d38ac6f5dd5"
server_key_iv: "73b151d9046bc2e143c78ea159dde2f6"
server_key_version: "encryptionkey@0"
I20260809 06:23:55.516067 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "4a3d7c9b40834939b451c7a2ba1ac974"
format_stamp: "Formatted at 2026-08-09 06:23:55 on dist-test-slave-2w3w"
server_key: "06318bc5af7d86e8d36c0d38ac6f5dd5"
server_key_iv: "73b151d9046bc2e143c78ea159dde2f6"
server_key_version: "encryptionkey@0"
I20260809 06:23:55.521106 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:23:55.524557 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:55.525880 23310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:23:55.526072 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "4a3d7c9b40834939b451c7a2ba1ac974"
format_stamp: "Formatted at 2026-08-09 06:23:55 on dist-test-slave-2w3w"
server_key: "06318bc5af7d86e8d36c0d38ac6f5dd5"
server_key_iv: "73b151d9046bc2e143c78ea159dde2f6"
server_key_version: "encryptionkey@0"
I20260809 06:23:55.526265 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:55.540275 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:55.544064 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:55.544343 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:55.545315 23310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:55.546654 23310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:55.546754 23310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:55.546897 23310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:55.546990 23310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 06:23:55.580404 23310 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:35141
I20260809 06:23:55.580451 23447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:35141 every 8 connection(s)
I20260809 06:23:55.581674 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:55.583586 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23310
I20260809 06:23:55.583807 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:23:55.584290 3621 external_mini_cluster.cc:1428] Setting key 2c1ba1ef8557acc2f9462712864577ff
Aug 09 06:23:55 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:55.601835 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:55.584421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:35345 (local address 127.3.137.126:36299)
0809 06:23:55.584969 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:55.584974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:55.586166 (+ 1192us) server_negotiation.cc:407] Connection header received
0809 06:23:55.587994 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:55.588000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:55.588086 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:55.588219 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:55.589705 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:55.590460 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:55.592088 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:55.592515 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:55.592657 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:55.596150 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:55.596155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:55.596158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:55.596191 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:55.598757 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:55.599521 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:55.599526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:55.599528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:55.599649 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:55.600099 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:55.600104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:55.600106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:55.600366 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:55.600544 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:55.601235 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 06:23:55.601478 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
I20260809 06:23:55.602777 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:55.585040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:36299 (local address 127.3.137.65:35345)
0809 06:23:55.585911 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:55.585967 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:55.587732 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:55.588456 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:55.588464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:55.588978 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:55.589532 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:55.589549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:55.590643 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:55.590647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:55.591934 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:55.591946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:55.592757 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:55.592765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:55.593008 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:55.593697 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:55.593719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:55.595916 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:55.598908 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:55.598918 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:55.598933 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:55.599375 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:55.599765 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:55.599770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:55.599774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:55.599970 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:55.600507 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:55.600514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:55.600839 (+ 325us) client_negotiation.cc:769] Sending connection context
0809 06:23:55.601109 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 06:23:55.601620 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":367,"threads_started":1}
I20260809 06:23:55.604419 23448 heartbeater.cc:347] Connected to a master server at 127.3.137.126:36299
I20260809 06:23:55.604766 23448 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:55.605705 23448 heartbeater.cc:511] Master 127.3.137.126:36299 requested a full tablet report, sending...
I20260809 06:23:55.608143 23245 ts_manager.cc:194] Registered new tserver with Master: 4a3d7c9b40834939b451c7a2ba1ac974 (127.3.137.65:35141)
I20260809 06:23:55.610307 23245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:35345
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:23:55.653981 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:55.979143 23454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:55.979408 23454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:55.979503 23454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:55.986562 23454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:55.986665 23454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:55.986843 23454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:55.996135 23454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:55.997282 23454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:55.998528 23454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:56.011801 23460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:56.011852 23463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:56.011875 23461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:56.013024 23454 server_base.cc:1034] running on GCE node
I20260809 06:23:56.013629 23454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:56.014532 23454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:56.015805 23454 hybrid_clock.cc:648] HybridClock initialized: now 1786256636015731 us; error 74 us; skew 500 ppm
I20260809 06:23:56.016162 23454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:56 dist-test-slave-2w3w krb5kdc[23195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:56.025265 23454 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:23:56.026767 23454 webserver.cc:460] Webserver started at http://127.3.137.66:38059/ using document root <none> and password file <none>
I20260809 06:23:56.027518 23454 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:56.027666 23454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:56.027994 23454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:56.031109 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "4b37cf686bae46d39761aa310d118f5a"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "a9ff7182bbbad27522889cb32ee427d2"
server_key_iv: "098dd98d19df84f8138c6b9c2873731e"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.032001 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "4b37cf686bae46d39761aa310d118f5a"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "a9ff7182bbbad27522889cb32ee427d2"
server_key_iv: "098dd98d19df84f8138c6b9c2873731e"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.037341 23454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:23:56.040804 23470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.042161 23454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:23:56.042366 23454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "4b37cf686bae46d39761aa310d118f5a"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "a9ff7182bbbad27522889cb32ee427d2"
server_key_iv: "098dd98d19df84f8138c6b9c2873731e"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.042594 23454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:56.066648 23454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:56.070897 23454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:56.071234 23454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:56.072266 23454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:56.073659 23454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:56.073791 23454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.073946 23454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:56.074029 23454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.110719 23454 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:39263
I20260809 06:23:56.110782 23595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:39263 every 8 connection(s)
I20260809 06:23:56.112201 23454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:56.121688 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23454
I20260809 06:23:56.121897 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:23:56.122503 3621 external_mini_cluster.cc:1428] Setting key 83d55ba89190f85f08a2b69904ce0df8
Aug 09 06:23:56 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:56.140524 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.115365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:51149 (local address 127.3.137.126:36299)
0809 06:23:56.115991 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.115999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.117402 (+ 1403us) server_negotiation.cc:407] Connection header received
0809 06:23:56.120534 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.120541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.120634 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.120779 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:56.123094 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.124202 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.126513 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.127335 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.127477 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.132789 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:56.132797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:56.132802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:56.132844 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:56.136599 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.137612 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.137620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.137626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.137806 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.138338 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.138345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.138351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.138732 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:56.138873 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.139912 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.140145 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":304,"threads_started":1}
I20260809 06:23:56.141556 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.116260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:36299 (local address 127.3.137.66:51149)
0809 06:23:56.117161 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.117224 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.120136 (+ 2912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.121033 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.121049 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.121937 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:56.122793 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.122823 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.124402 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.124408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.126320 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.126345 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.127808 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.127818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.128150 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.128992 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:56.129033 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:56.132520 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:56.136776 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:56.136788 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:56.136807 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:56.137439 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:56.137939 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:56.137946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:56.137952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:56.138181 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:56.138996 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:56.139008 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:56.139516 (+ 508us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.139827 (+ 311us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.140339 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":308,"threads_started":1}
I20260809 06:23:56.143395 23596 heartbeater.cc:347] Connected to a master server at 127.3.137.126:36299
I20260809 06:23:56.143693 23596 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:56.144455 23596 heartbeater.cc:511] Master 127.3.137.126:36299 requested a full tablet report, sending...
I20260809 06:23:56.145988 23245 ts_manager.cc:194] Registered new tserver with Master: 4b37cf686bae46d39761aa310d118f5a (127.3.137.66:39263)
I20260809 06:23:56.147066 23245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:51149
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:23:56.202919 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:23:56.533155 23604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:56.533424 23604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:56.533519 23604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:56.540872 23604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:56.540972 23604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:56.541092 23604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:56.550812 23604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:56.552181 23604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:56.553404 23604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:56.566612 23610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:56.566671 23613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:56.566700 23611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:56.567643 23604 server_base.cc:1034] running on GCE node
I20260809 06:23:56.568186 23604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:56.569021 23604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:56.570216 23604 hybrid_clock.cc:648] HybridClock initialized: now 1786256636570183 us; error 29 us; skew 500 ppm
I20260809 06:23:56.570554 23604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:23:56 dist-test-slave-2w3w krb5kdc[23195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:23:56.573660 23604 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:23:56.575151 23604 webserver.cc:460] Webserver started at http://127.3.137.67:33259/ using document root <none> and password file <none>
I20260809 06:23:56.575814 23604 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:56.575958 23604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:56.576269 23604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:56.579097 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "1f79972306684169baa77a29dc80aff7"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "d1c066563039ebc1b95244b68689a651"
server_key_iv: "031cc70ee40f75dcb6afaaefce04b590"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.579914 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "1f79972306684169baa77a29dc80aff7"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "d1c066563039ebc1b95244b68689a651"
server_key_iv: "031cc70ee40f75dcb6afaaefce04b590"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.584965 23604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 06:23:56.588680 23620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.589946 23604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:23:56.590131 23604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "1f79972306684169baa77a29dc80aff7"
format_stamp: "Formatted at 2026-08-09 06:23:56 on dist-test-slave-2w3w"
server_key: "d1c066563039ebc1b95244b68689a651"
server_key_iv: "031cc70ee40f75dcb6afaaefce04b590"
server_key_version: "encryptionkey@0"
I20260809 06:23:56.590334 23604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:56.613729 23604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:56.614209 23448 heartbeater.cc:503] Master 127.3.137.126:36299 was elected leader, sending a full tablet report...
I20260809 06:23:56.618176 23604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:56.618423 23604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:56.619392 23604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:56.620656 23604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:56.620726 23604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.620826 23604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:56.620869 23604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:56.656487 23604 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:46691
I20260809 06:23:56.656589 23743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:46691 every 8 connection(s)
I20260809 06:23:56.657764 23604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:56.662196 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23604
I20260809 06:23:56.662408 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:23:56.663017 3621 external_mini_cluster.cc:1428] Setting key fbea4c7c1a13c1eb93786e9caca38c7b
Aug 09 06:23:56 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:23:56.678560 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.660204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:52415 (local address 127.3.137.126:36299)
0809 06:23:56.660631 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.660636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.661562 (+ 926us) server_negotiation.cc:407] Connection header received
0809 06:23:56.664111 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.664115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.664197 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.664333 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:56.666354 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.667234 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.668853 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.669266 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.669371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.672709 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:56.672713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:56.672716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:56.672754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:56.675395 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.676065 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.676069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.676071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.676195 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.676603 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.676606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.676609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.676966 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:56.677105 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.677961 (+ 856us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.678134 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
I20260809 06:23:56.679554 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.660734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:36299 (local address 127.3.137.67:52415)
0809 06:23:56.661384 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.661429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.663794 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.664606 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.664613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.665376 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:56.666200 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.666221 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.667378 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.667381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.668654 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.668666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.669467 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.669472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.669663 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.670360 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:23:56.670382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:56.672526 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:56.675527 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:56.675532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:56.675543 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:56.675947 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:56.676294 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:23:56.676297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:23:56.676299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:23:56.676480 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:23:56.677116 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:56.677125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:56.677511 (+ 386us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.677831 (+ 320us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.678348 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260809 06:23:56.680924 23744 heartbeater.cc:347] Connected to a master server at 127.3.137.126:36299
I20260809 06:23:56.681226 23744 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:56.681896 23744 heartbeater.cc:511] Master 127.3.137.126:36299 requested a full tablet report, sending...
I20260809 06:23:56.683331 23245 ts_manager.cc:194] Registered new tserver with Master: 1f79972306684169baa77a29dc80aff7 (127.3.137.67:46691)
I20260809 06:23:56.684162 23245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:52415
I20260809 06:23:56.690701 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:23:56.708684 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.695303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.3.137.126:36299)
0809 06:23:56.695515 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.695524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.695940 (+ 416us) server_negotiation.cc:407] Connection header received
0809 06:23:56.696049 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.696052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.696108 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.696186 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:56.697567 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.698298 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.699855 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.700274 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.700368 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.703555 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:56.703559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:56.703562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:56.703592 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:56.706023 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.706808 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.706811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.706814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.706922 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:56.707368 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:56.707371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:56.707373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:56.707653 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:56.707844 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.708161 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.708303 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:23:56.713261 23245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:23:56.720114 23245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:23:56.758015 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.732166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.3.137.66:39263)
0809 06:23:56.747602 (+ 15436us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.747614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.747651 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:23:56.747788 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.747800 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.748118 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.749794 (+ 1676us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:56.750075 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.751717 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.755072 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.756598 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.756750 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.756938 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.757436 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.757614 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15203,"thread_start_us":16540,"threads_started":1}
I20260809 06:23:56.758136 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.731970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:39263 (local address 127.0.0.1:43662)
0809 06:23:56.741548 (+ 9578us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.741588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.741753 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.749024 (+ 7271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.749031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.749061 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:56.749587 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.749605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.751948 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.751953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.754735 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.754849 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.756819 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.756826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.757085 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.757147 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.757297 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.757546 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9340,"mutex_wait_us":99,"thread_start_us":9950,"threads_started":1}
I20260809 06:23:56.758874 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.732515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.3.137.65:35141)
0809 06:23:56.748836 (+ 16321us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.748844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.748869 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:23:56.748971 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.748978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.749201 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.749401 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:56.750314 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.751865 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.755174 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.756598 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.756739 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.756969 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.758367 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.758545 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16152,"thread_start_us":15998,"threads_started":1}
I20260809 06:23:56.759133 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.732203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:35141 (local address 127.0.0.1:45882)
0809 06:23:56.742148 (+ 9945us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.742182 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.742293 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.749640 (+ 7347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.749644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.749670 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:56.750094 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.750111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.752001 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.752008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.755006 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.755021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.757183 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.757189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.757396 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.757444 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.757550 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.757685 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9812,"thread_start_us":8939,"threads_started":1}
I20260809 06:23:56.762513 23379 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 (DEFAULT_TABLE table=test-table [id=1e92d1918fa447b4ba9dfc072d181be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:56.762506 23525 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 (DEFAULT_TABLE table=test-table [id=1e92d1918fa447b4ba9dfc072d181be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:56.763379 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.742511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:46691 (local address 127.0.0.1:58798)
0809 06:23:56.753725 (+ 11214us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.753764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.754055 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.755093 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.755098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.755121 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:56.755433 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.755441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.756824 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.756828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.759621 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.759636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.762529 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.762535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.762704 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.762744 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.762852 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.763001 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10954,"mutex_wait_us":34,"thread_start_us":11378,"threads_started":1}
I20260809 06:23:56.763785 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.742626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.3.137.67:46691)
0809 06:23:56.753098 (+ 10472us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.753106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.753911 (+ 805us) server_negotiation.cc:407] Connection header received
0809 06:23:56.754267 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.754276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.754487 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.754682 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:56.755597 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.756641 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.759788 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.762318 (+ 2530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.762481 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.762711 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.762906 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.763105 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10301,"thread_start_us":10134,"threads_started":1}
I20260809 06:23:56.764041 23379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6213fb05246999064a84cc6a3c1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:56.764070 23525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6213fb05246999064a84cc6a3c1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:56.766746 23672 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 (DEFAULT_TABLE table=test-table [id=1e92d1918fa447b4ba9dfc072d181be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:56.768090 23672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6213fb05246999064a84cc6a3c1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:56.778395 23764 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Bootstrap starting.
I20260809 06:23:56.782915 23766 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Bootstrap starting.
I20260809 06:23:56.783635 23765 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Bootstrap starting.
I20260809 06:23:56.785952 23766 tablet_bootstrap.cc:654] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:56.786314 23765 tablet_bootstrap.cc:654] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:56.787106 23766 log.cc:824] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:56.787398 23765 log.cc:824] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:56.787827 23764 tablet_bootstrap.cc:654] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:56.789090 23764 log.cc:824] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:56.790354 23766 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: No bootstrap required, opened a new log
I20260809 06:23:56.790635 23766 ts_tablet_manager.cc:1397] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:23:56.791452 23765 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: No bootstrap required, opened a new log
I20260809 06:23:56.791713 23765 ts_tablet_manager.cc:1397] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:23:56.792730 23764 tablet_bootstrap.cc:492] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: No bootstrap required, opened a new log
I20260809 06:23:56.792997 23764 ts_tablet_manager.cc:1397] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260809 06:23:56.798666 23766 raft_consensus.cc:348] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.799168 23766 raft_consensus.cc:374] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:56.799270 23766 raft_consensus.cc:729] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f79972306684169baa77a29dc80aff7, State: Initialized, Role: FOLLOWER
I20260809 06:23:56.799880 23766 consensus_queue.cc:260] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.800976 23744 heartbeater.cc:503] Master 127.3.137.126:36299 was elected leader, sending a full tablet report...
I20260809 06:23:56.801649 23766 ts_tablet_manager.cc:1428] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 06:23:56.802376 23765 raft_consensus.cc:348] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.802788 23765 raft_consensus.cc:374] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:56.802886 23765 raft_consensus.cc:729] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b37cf686bae46d39761aa310d118f5a, State: Initialized, Role: FOLLOWER
I20260809 06:23:56.803494 23765 consensus_queue.cc:260] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.803540 23764 raft_consensus.cc:348] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.803947 23764 raft_consensus.cc:374] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:56.804075 23764 raft_consensus.cc:729] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3d7c9b40834939b451c7a2ba1ac974, State: Initialized, Role: FOLLOWER
I20260809 06:23:56.804647 23596 heartbeater.cc:503] Master 127.3.137.126:36299 was elected leader, sending a full tablet report...
I20260809 06:23:56.804719 23764 consensus_queue.cc:260] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.805370 23765 ts_tablet_manager.cc:1428] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 06:23:56.807044 23764 ts_tablet_manager.cc:1428] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
W20260809 06:23:56.834189 23449 tablet.cc:2369] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:23:56.864444 23598 tablet.cc:2369] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:23:56.888147 23771 raft_consensus.cc:482] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:56.888484 23771 raft_consensus.cc:504] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.890158 23771 leader_election.cc:288] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a3d7c9b40834939b451c7a2ba1ac974 (127.3.137.65:35141), 1f79972306684169baa77a29dc80aff7 (127.3.137.67:46691)
I20260809 06:23:56.898360 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.890599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:33177 (local address 127.3.137.67:46691)
0809 06:23:56.890829 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.890838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.891560 (+ 722us) server_negotiation.cc:407] Connection header received
0809 06:23:56.891741 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.891746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.891832 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.891930 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:56.892644 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.893983 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.896365 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.897332 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.897502 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.897604 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.897910 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.898060 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:23:56.898384 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.890517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:46691 (local address 127.3.137.66:33177)
0809 06:23:56.891445 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.891461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.891576 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.891952 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.891956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.891981 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:56.892469 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.892480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.894104 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.894110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.896196 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.896208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.897574 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.897580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.897741 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.897767 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.897848 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.897953 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":78,"thread_start_us":579,"threads_started":1}
I20260809 06:23:56.898391 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.890504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:50893 (local address 127.3.137.65:35141)
0809 06:23:56.890725 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.890731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.891756 (+ 1025us) server_negotiation.cc:407] Connection header received
0809 06:23:56.891938 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.891944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.892023 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.892122 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:56.892923 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.894101 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.896593 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.897366 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.897561 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.897632 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.898012 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.898149 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:23:56.898810 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.890558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:35141 (local address 127.3.137.66:50893)
0809 06:23:56.891611 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:56.891649 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:56.891781 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:56.892180 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:56.892185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:56.892210 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:56.892697 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.892712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.894250 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.894255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.896432 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:56.896448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.897638 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.897645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:56.897807 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.897866 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 06:23:56.897972 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:23:56.898122 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":3,"thread_start_us":612,"threads_started":1}
I20260809 06:23:56.899454 23693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6213fb05246999064a84cc6a3c1c2" candidate_uuid: "4b37cf686bae46d39761aa310d118f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f79972306684169baa77a29dc80aff7" is_pre_election: true
I20260809 06:23:56.900032 23693 raft_consensus.cc:2454] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b37cf686bae46d39761aa310d118f5a in term 0.
I20260809 06:23:56.900009 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6213fb05246999064a84cc6a3c1c2" candidate_uuid: "4b37cf686bae46d39761aa310d118f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" is_pre_election: true
I20260809 06:23:56.900660 23400 raft_consensus.cc:2454] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b37cf686bae46d39761aa310d118f5a in term 0.
I20260809 06:23:56.900735 23474 leader_election.cc:302] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f79972306684169baa77a29dc80aff7, 4b37cf686bae46d39761aa310d118f5a; no voters:
I20260809 06:23:56.901176 23771 raft_consensus.cc:2781] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:23:56.901331 23771 raft_consensus.cc:482] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:23:56.901469 23771 raft_consensus.cc:3037] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:56.903678 23771 raft_consensus.cc:504] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.904266 23771 leader_election.cc:288] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [CANDIDATE]: Term 1 election: Requested vote from peers 4a3d7c9b40834939b451c7a2ba1ac974 (127.3.137.65:35141), 1f79972306684169baa77a29dc80aff7 (127.3.137.67:46691)
I20260809 06:23:56.904606 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6213fb05246999064a84cc6a3c1c2" candidate_uuid: "4b37cf686bae46d39761aa310d118f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3d7c9b40834939b451c7a2ba1ac974"
I20260809 06:23:56.904767 23693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6213fb05246999064a84cc6a3c1c2" candidate_uuid: "4b37cf686bae46d39761aa310d118f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f79972306684169baa77a29dc80aff7"
I20260809 06:23:56.904814 23400 raft_consensus.cc:3037] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:56.904989 23693 raft_consensus.cc:3037] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:56.908097 23400 raft_consensus.cc:2454] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b37cf686bae46d39761aa310d118f5a in term 1.
I20260809 06:23:56.908151 23693 raft_consensus.cc:2454] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b37cf686bae46d39761aa310d118f5a in term 1.
I20260809 06:23:56.908571 23475 leader_election.cc:302] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3d7c9b40834939b451c7a2ba1ac974, 4b37cf686bae46d39761aa310d118f5a; no voters:
I20260809 06:23:56.908864 23771 raft_consensus.cc:2781] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:56.909385 23771 raft_consensus.cc:686] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 1 LEADER]: Becoming Leader. State: Replica: 4b37cf686bae46d39761aa310d118f5a, State: Running, Role: LEADER
W20260809 06:23:56.909399 23745 tablet.cc:2369] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:23:56.909859 23771 consensus_queue.cc:237] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } }
I20260809 06:23:56.914319 23245 catalog_manager.cc:5495] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b37cf686bae46d39761aa310d118f5a (127.3.137.66). New cstate: current_term: 1 leader_uuid: "4b37cf686bae46d39761aa310d118f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b37cf686bae46d39761aa310d118f5a" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 } health_report { overall_health: UNKNOWN } } }
I20260809 06:23:56.962600 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:56.956730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.3.137.66:39263)
0809 06:23:56.956969 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:56.956975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:56.957103 (+ 128us) server_negotiation.cc:407] Connection header received
0809 06:23:56.957360 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:56.957364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:56.957450 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:56.957602 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:56.958037 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.958752 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.960281 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:56.960774 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:56.960918 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:56.961423 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:56.962065 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:23:56.962199 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:56.962333 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 06:23:56.962372 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:23:56.971091 23693 raft_consensus.cc:1261] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 1 FOLLOWER]: Refusing update from remote peer 4b37cf686bae46d39761aa310d118f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:56.971123 23400 raft_consensus.cc:1261] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 1 FOLLOWER]: Refusing update from remote peer 4b37cf686bae46d39761aa310d118f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:56.972131 23775 consensus_queue.cc:1035] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3d7c9b40834939b451c7a2ba1ac974" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 35141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:56.972458 23771 consensus_queue.cc:1035] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79972306684169baa77a29dc80aff7" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 46691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:56.997941 23779 mvcc.cc:205] Tried to move back new op lower bound from 7316507185023295488 to 7316507184786223104. Current Snapshot: MvccSnapshot[applied={T|T < 7316507185023295488}]
I20260809 06:23:57.012782 23245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:57.013080 23245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:57.017308 23245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71208be4afad4fb8b40374470440081b: Sending DeleteTablet for 3 replicas of tablet 10c6213fb05246999064a84cc6a3c1c2
I20260809 06:23:57.018399 23525 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:43662
I20260809 06:23:57.019292 23379 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:45882
I20260809 06:23:57.020040 23672 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6213fb05246999064a84cc6a3c1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:57 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:58798
I20260809 06:23:57.020864 23789 tablet_replica.cc:330] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: stopping tablet replica
I20260809 06:23:57.021338 23789 raft_consensus.cc:2229] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:57.021909 23789 raft_consensus.cc:2258] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:57.023502 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23310
I20260809 06:23:57.025120 23789 ts_tablet_manager.cc:1905] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:57.032074 23787 tablet_replica.cc:330] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: stopping tablet replica
I20260809 06:23:57.032493 23787 raft_consensus.cc:2229] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:23:57.033177 23787 raft_consensus.cc:2258] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:57.035385 23789 ts_tablet_manager.cc:1918] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:57.035593 23789 log.cc:1196] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/10c6213fb05246999064a84cc6a3c1c2
I20260809 06:23:57.035890 23787 ts_tablet_manager.cc:1905] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:57.036265 23789 ts_tablet_manager.cc:1939] T 10c6213fb05246999064a84cc6a3c1c2 P 1f79972306684169baa77a29dc80aff7: Deleting consensus metadata
I20260809 06:23:57.038399 23230 catalog_manager.cc:4841] TS 1f79972306684169baa77a29dc80aff7 (127.3.137.67:46691): tablet 10c6213fb05246999064a84cc6a3c1c2 (table test-table [id=1e92d1918fa447b4ba9dfc072d181be3]) successfully deleted
I20260809 06:23:57.040693 23788 tablet_replica.cc:330] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: stopping tablet replica
I20260809 06:23:57.041123 23788 raft_consensus.cc:2229] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:57.041689 23788 raft_consensus.cc:2258] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:57.043993 23788 ts_tablet_manager.cc:1905] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:57.044646 23787 ts_tablet_manager.cc:1918] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:57.044870 23787 log.cc:1196] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/10c6213fb05246999064a84cc6a3c1c2
I20260809 06:23:57.045471 23787 ts_tablet_manager.cc:1939] T 10c6213fb05246999064a84cc6a3c1c2 P 4b37cf686bae46d39761aa310d118f5a: Deleting consensus metadata
I20260809 06:23:57.049336 23230 catalog_manager.cc:4841] TS 4b37cf686bae46d39761aa310d118f5a (127.3.137.66:39263): tablet 10c6213fb05246999064a84cc6a3c1c2 (table test-table [id=1e92d1918fa447b4ba9dfc072d181be3]) successfully deleted
I20260809 06:23:57.051608 23788 ts_tablet_manager.cc:1918] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:57.051779 23788 log.cc:1196] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/10c6213fb05246999064a84cc6a3c1c2
I20260809 06:23:57.052338 23788 ts_tablet_manager.cc:1939] T 10c6213fb05246999064a84cc6a3c1c2 P 4a3d7c9b40834939b451c7a2ba1ac974: Deleting consensus metadata
I20260809 06:23:57.054096 23231 catalog_manager.cc:4841] TS 4a3d7c9b40834939b451c7a2ba1ac974 (127.3.137.65:35141): tablet 10c6213fb05246999064a84cc6a3c1c2 (table test-table [id=1e92d1918fa447b4ba9dfc072d181be3]) successfully deleted
Aug 09 06:23:57 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:23:57.064198 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:57.024641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.3.137.65:35141)
0809 06:23:57.024865 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:57.024874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:57.039849 (+ 14975us) server_negotiation.cc:407] Connection header received
0809 06:23:57.040063 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:57.040068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:57.040150 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:57.040263 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:57.042253 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.043414 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.051254 (+ 7840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.053356 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.053522 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:57.058461 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:57.058472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:57.058486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:57.058516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:57.061341 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.062042 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.062049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.062051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.062180 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.062578 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.062582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.062584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.062922 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:57.063285 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:57.063725 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 06:23:57.063884 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:23:57.065017 23442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:57.157147 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23310
I20260809 06:23:57.184530 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23454
Aug 09 06:23:57 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:23:57.201025 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:57.185398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.3.137.66:39263)
0809 06:23:57.185648 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:57.185653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:57.185673 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:23:57.185832 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:57.185841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:57.185906 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:57.186016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:57.187398 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.188249 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.189525 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.189978 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.190118 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:57.194748 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:57.194760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:57.194776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:57.194817 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:57.197932 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.198635 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.198643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.198645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.198789 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.199339 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.199347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.199350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.199738 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:57.200087 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:57.200393 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 06:23:57.200636 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 06:23:57.201846 23590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:57.312906 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23454
I20260809 06:23:57.332955 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23604
Aug 09 06:23:57 dist-test-slave-2w3w krb5kdc[23195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:23:57.347705 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:57.333761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.3.137.67:46691)
0809 06:23:57.333950 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:57.333956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:57.333976 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:23:57.334088 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:57.334093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:57.334155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:57.334237 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:57.335623 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.336353 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.337509 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:57.337891 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:57.337984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:57.342565 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:57.342578 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:57.342593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:23:57.342625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:57.345203 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.345830 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.345837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.345839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.345972 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:23:57.346283 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:23:57.346286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:23:57.346288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:23:57.346668 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:57.346907 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:57.347200 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 06:23:57.347353 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:23:57.348349 23737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:57.438441 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23604
I20260809 06:23:57.458098 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23209
I20260809 06:23:57.459267 23276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:57.569391 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23209
2026-08-09T06:23:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4489 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T06:23:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:23:57Z Disabled control of system clock
I20260809 06:23:57.655297 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39107
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:23:57.984663 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:57.984937 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:57.985052 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:57.992298 23800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:23:57.992401 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:57.992496 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:57.992596 23800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:23:57.992683 23800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:23:58.001806 23800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:58.003140 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:58.004971 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:58.016906 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:58.017002 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:58.016975 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:58.018476 23800 server_base.cc:1034] running on GCE node
I20260809 06:23:58.019301 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:58.020612 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:58.021814 23800 hybrid_clock.cc:648] HybridClock initialized: now 1786256638021771 us; error 33 us; skew 500 ppm
I20260809 06:23:58.022171 23800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:58.024464 23800 webserver.cc:460] Webserver started at http://127.3.137.126:39083/ using document root <none> and password file <none>
I20260809 06:23:58.025137 23800 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:58.025254 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:58.025602 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:58.028568 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "2b86407f0af245cd96977703b9c613ba"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "c573f51ee5af5ddf71bf6318a0732fac"
server_key_iv: "f790d6229cfbe9e8ee186b12c5baea2c"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.029354 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "2b86407f0af245cd96977703b9c613ba"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "c573f51ee5af5ddf71bf6318a0732fac"
server_key_iv: "f790d6229cfbe9e8ee186b12c5baea2c"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.034245 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:23:58.037696 23817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:58.039011 23800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:58.039193 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "2b86407f0af245cd96977703b9c613ba"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "c573f51ee5af5ddf71bf6318a0732fac"
server_key_iv: "f790d6229cfbe9e8ee186b12c5baea2c"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.039404 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:58.055199 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:58.055962 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:58.056246 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:58.074651 23800 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:39107
I20260809 06:23:58.074684 23875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:39107 every 8 connection(s)
I20260809 06:23:58.076001 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:23:58.079030 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23800
I20260809 06:23:58.079237 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:23:58.079650 23876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:58.079712 3621 external_mini_cluster.cc:1428] Setting key ef59df34cf8577f55b9549328a590586
I20260809 06:23:58.094651 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:58.081525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.3.137.126:39107)
0809 06:23:58.082619 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:58.082639 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:58.082692 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:23:58.084627 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:58.084657 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:58.085247 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:58.085681 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:58.086229 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.088302 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.089612 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.090657 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.090852 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:58.092003 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:58.092013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:58.092030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:58.092752 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:58.092841 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:58.092870 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:58.092991 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:58.093158 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:23:58.093457 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":357,"threads_started":1}
I20260809 06:23:58.094870 23876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: Bootstrap starting.
I20260809 06:23:58.097894 23876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:58.098860 23876 log.cc:824] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:58.101380 23876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: No bootstrap required, opened a new log
I20260809 06:23:58.108502 23876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } }
I20260809 06:23:58.108870 23876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:58.108996 23876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b86407f0af245cd96977703b9c613ba, State: Initialized, Role: FOLLOWER
I20260809 06:23:58.109560 23876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } }
I20260809 06:23:58.109797 23876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:23:58.109953 23876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:23:58.110124 23876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:58.112341 23876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } }
I20260809 06:23:58.112874 23876 leader_election.cc:302] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b86407f0af245cd96977703b9c613ba; no voters:
I20260809 06:23:58.113448 23876 leader_election.cc:288] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:23:58.113591 23881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:58.114328 23881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [term 1 LEADER]: Becoming Leader. State: Replica: 2b86407f0af245cd96977703b9c613ba, State: Running, Role: LEADER
I20260809 06:23:58.114876 23881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } }
I20260809 06:23:58.115664 23876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:23:58.117532 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b86407f0af245cd96977703b9c613ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } } }
I20260809 06:23:58.117854 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:58.118530 23883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b86407f0af245cd96977703b9c613ba. Latest consensus state: current_term: 1 leader_uuid: "2b86407f0af245cd96977703b9c613ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b86407f0af245cd96977703b9c613ba" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39107 } } }
I20260809 06:23:58.118788 23883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba [sys.catalog]: This master's current role is: LEADER
I20260809 06:23:58.119453 23887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:23:58.125787 23887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:23:58.134521 23887 catalog_manager.cc:1347] Generated new cluster ID: 1e7fc6338e514de19baca60d93766eff
I20260809 06:23:58.134642 23887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:23:58.147202 23887 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:23:58.148518 23887 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:23:58.162077 23887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: Generated new TSK 0
I20260809 06:23:58.162849 23887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:23:58.175510 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--builtin_ntp_servers=127.3.137.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:23:58.530354 23902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:58.530620 23902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:58.530706 23902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:58.537828 23902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:58.537915 23902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:58.538024 23902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:23:58.546981 23902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:58.548111 23902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:58.549803 23902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:58.562696 23908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:58.562733 23909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:58.562784 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:58.563457 23902 server_base.cc:1034] running on GCE node
I20260809 06:23:58.564042 23902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:58.564922 23902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:58.566138 23902 hybrid_clock.cc:648] HybridClock initialized: now 1786256638566080 us; error 50 us; skew 500 ppm
I20260809 06:23:58.566461 23902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:58.568737 23902 webserver.cc:460] Webserver started at http://127.3.137.65:39697/ using document root <none> and password file <none>
I20260809 06:23:58.569402 23902 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:58.569523 23902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:58.569844 23902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:58.572724 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "c6c951690e3d400a9d6a6489e6e3454f"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "866e73386711633fd0b6dc57a05843a9"
server_key_iv: "3049c25a0b8271adce449573f90d4f7a"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.573524 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c951690e3d400a9d6a6489e6e3454f"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "866e73386711633fd0b6dc57a05843a9"
server_key_iv: "3049c25a0b8271adce449573f90d4f7a"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.578321 23902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260809 06:23:58.581622 23918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:58.582847 23902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 06:23:58.583073 23902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "c6c951690e3d400a9d6a6489e6e3454f"
format_stamp: "Formatted at 2026-08-09 06:23:58 on dist-test-slave-2w3w"
server_key: "866e73386711633fd0b6dc57a05843a9"
server_key_iv: "3049c25a0b8271adce449573f90d4f7a"
server_key_version: "encryptionkey@0"
I20260809 06:23:58.583292 23902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:58.597254 23902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:58.598030 23902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:58.598337 23902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:58.599355 23902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:58.600723 23902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:58.600857 23902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:58.600999 23902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:58.601087 23902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:58.636217 23902 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:44897
I20260809 06:23:58.636276 24040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:44897 every 8 connection(s)
I20260809 06:23:58.637521 23902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:23:58.640348 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 23902
I20260809 06:23:58.640520 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:23:58.641081 3621 external_mini_cluster.cc:1428] Setting key ac4459124d3b4915fa9cf67d8a726983
I20260809 06:23:58.644335 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--builtin_ntp_servers=127.3.137.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:23:58.650566 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:58.640122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:60705 (local address 127.3.137.126:39107)
0809 06:23:58.640581 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:58.640585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:58.641999 (+ 1414us) server_negotiation.cc:407] Connection header received
0809 06:23:58.644315 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:58.644321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:58.644415 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:58.644531 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:58.645588 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.646326 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.647929 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.648322 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.648450 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:58.649754 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:58.649759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:58.649762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:58.649781 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:58.649818 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:58.649829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:58.649924 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:58.650126 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 06:23:58.650219 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":266,"threads_started":1}
I20260809 06:23:58.651543 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:58.640802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39107 (local address 127.3.137.65:60705)
0809 06:23:58.641840 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:58.641897 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:58.644041 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:58.644813 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:58.644821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:58.644898 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:58.645426 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:58.645442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.646482 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.646487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:58.647792 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:58.647804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:58.648545 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:58.648553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:58.648781 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:58.649455 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:58.649458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:58.649494 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:58.649562 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:58.649921 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:58.649932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:58.649941 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 06:23:58.650116 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 06:23:58.650474 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":340,"threads_started":1}
I20260809 06:23:58.653057 24041 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39107
I20260809 06:23:58.653335 24041 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:58.654024 24041 heartbeater.cc:511] Master 127.3.137.126:39107 requested a full tablet report, sending...
I20260809 06:23:58.656075 23838 ts_manager.cc:194] Registered new tserver with Master: c6c951690e3d400a9d6a6489e6e3454f (127.3.137.65:44897)
I20260809 06:23:58.657500 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.65:60705
W20260809 06:23:58.976574 24046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:58.976825 24046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:58.976956 24046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:58.984028 24046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:58.984119 24046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:58.984319 24046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:23:58.993085 24046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:58.994313 24046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:58.996112 24046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:59.008915 24051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:59.008976 24055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:59.008976 24053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:59.009980 24046 server_base.cc:1034] running on GCE node
I20260809 06:23:59.010550 24046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:59.011529 24046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:59.012728 24046 hybrid_clock.cc:648] HybridClock initialized: now 1786256639012672 us; error 51 us; skew 500 ppm
I20260809 06:23:59.013054 24046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:59.015292 24046 webserver.cc:460] Webserver started at http://127.3.137.66:37107/ using document root <none> and password file <none>
I20260809 06:23:59.015991 24046 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:59.016129 24046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:59.016475 24046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:59.019441 24046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "b492c5c83c90441e9511dce77c61bc81"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "683aabbd3821e306a721bbd8a5a74a05"
server_key_iv: "3e3d5b952e987e6031dd9c66c6880d92"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.020262 24046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "b492c5c83c90441e9511dce77c61bc81"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "683aabbd3821e306a721bbd8a5a74a05"
server_key_iv: "3e3d5b952e987e6031dd9c66c6880d92"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.025413 24046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:23:59.028929 24061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.030164 24046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:23:59.030344 24046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "b492c5c83c90441e9511dce77c61bc81"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "683aabbd3821e306a721bbd8a5a74a05"
server_key_iv: "3e3d5b952e987e6031dd9c66c6880d92"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.030568 24046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:59.051424 24046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:59.052196 24046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:59.052506 24046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:59.053531 24046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:59.054919 24046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:59.055069 24046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.055197 24046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:59.055286 24046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.094892 24046 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:43331
I20260809 06:23:59.095044 24183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:43331 every 8 connection(s)
I20260809 06:23:59.096199 24046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:23:59.104786 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24046
I20260809 06:23:59.104979 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:23:59.105412 3621 external_mini_cluster.cc:1428] Setting key 42108197120bc92c8d0b91f28f8d602f
I20260809 06:23:59.108197 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--builtin_ntp_servers=127.3.137.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:23:59.108933 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.098784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:36319 (local address 127.3.137.126:39107)
0809 06:23:59.099021 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.099026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.100345 (+ 1319us) server_negotiation.cc:407] Connection header received
0809 06:23:59.102153 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.102160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.102262 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.102377 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:59.103489 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.104208 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.105964 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.106506 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.106637 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.108118 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:59.108124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:59.108128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:59.108165 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:59.108196 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:59.108207 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:59.108314 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.108541 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.108636 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:23:59.109875 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.099330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39107 (local address 127.3.137.66:36319)
0809 06:23:59.100185 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.100222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.101878 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.102606 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.102614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.102693 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:59.103323 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.103340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.104394 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.104398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.105808 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.105821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.106741 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.106750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.107011 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.107782 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:59.107785 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:59.107826 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:59.107903 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:59.108313 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:59.108326 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:59.108337 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.108513 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.108887 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260809 06:23:59.111336 24184 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39107
I20260809 06:23:59.111629 24184 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:59.112429 24184 heartbeater.cc:511] Master 127.3.137.126:39107 requested a full tablet report, sending...
I20260809 06:23:59.113768 23838 ts_manager.cc:194] Registered new tserver with Master: b492c5c83c90441e9511dce77c61bc81 (127.3.137.66:43331)
I20260809 06:23:59.114602 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.66:36319
W20260809 06:23:59.439393 24187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:23:59.439657 24187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:23:59.439832 24187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:23:59.447127 24187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:23:59.447217 24187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:23:59.447417 24187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:23:59.456171 24187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:23:59.457337 24187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:23:59.459100 24187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:23:59.471824 24194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:59.471884 24197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:23:59.472124 24195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:23:59.473341 24187 server_base.cc:1034] running on GCE node
I20260809 06:23:59.473955 24187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:23:59.474813 24187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:23:59.476042 24187 hybrid_clock.cc:648] HybridClock initialized: now 1786256639475989 us; error 61 us; skew 500 ppm
I20260809 06:23:59.476377 24187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:23:59.478552 24187 webserver.cc:460] Webserver started at http://127.3.137.67:33755/ using document root <none> and password file <none>
I20260809 06:23:59.479301 24187 fs_manager.cc:359] Metadata directory not provided
I20260809 06:23:59.479432 24187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:23:59.479763 24187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:23:59.482501 24187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "1da870a7298e4e5aaf8bd5fb15efd369"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "df748a2ddd5959eb0109e5e52f0592ad"
server_key_iv: "5477d535a19bfa39b122cf72b52ebe7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.483351 24187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "1da870a7298e4e5aaf8bd5fb15efd369"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "df748a2ddd5959eb0109e5e52f0592ad"
server_key_iv: "5477d535a19bfa39b122cf72b52ebe7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.488368 24187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:23:59.491779 24204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.493011 24187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:23:59.493191 24187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "1da870a7298e4e5aaf8bd5fb15efd369"
format_stamp: "Formatted at 2026-08-09 06:23:59 on dist-test-slave-2w3w"
server_key: "df748a2ddd5959eb0109e5e52f0592ad"
server_key_iv: "5477d535a19bfa39b122cf72b52ebe7d"
server_key_version: "encryptionkey@0"
I20260809 06:23:59.493432 24187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:23:59.512554 24187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:23:59.513361 24187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:23:59.513665 24187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:23:59.514611 24187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:23:59.516266 24187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:23:59.516393 24187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.516530 24187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:23:59.516628 24187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:23:59.551852 24187 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:44199
I20260809 06:23:59.551941 24324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:44199 every 8 connection(s)
I20260809 06:23:59.553138 24187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:23:59.558648 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24187
I20260809 06:23:59.558830 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:23:59.559336 3621 external_mini_cluster.cc:1428] Setting key f55ea007f77373c12b23cfcf052fb887
I20260809 06:23:59.566160 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.555708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:37057 (local address 127.3.137.126:39107)
0809 06:23:59.555892 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.555897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.557202 (+ 1305us) server_negotiation.cc:407] Connection header received
0809 06:23:59.559255 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.559261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.559366 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.559479 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:59.560823 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.561716 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.563494 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.563902 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.564038 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.565322 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:59.565326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:59.565329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:59.565345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:59.565377 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:59.565400 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:59.565497 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.565736 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.565833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:23:59.567246 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.556176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39107 (local address 127.3.137.67:37057)
0809 06:23:59.557043 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.557084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.558951 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.559712 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.559719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.559856 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:23:59.560628 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.560656 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.561864 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.561869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.563354 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.563366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.564124 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.564131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.564369 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.564998 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:23:59.565000 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:23:59.565043 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:23:59.565113 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:23:59.565497 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:23:59.565510 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:23:59.565520 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.565716 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.566101 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260809 06:23:59.568476 24325 heartbeater.cc:347] Connected to a master server at 127.3.137.126:39107
I20260809 06:23:59.568756 24325 heartbeater.cc:464] Registering TS with master...
I20260809 06:23:59.569448 24325 heartbeater.cc:511] Master 127.3.137.126:39107 requested a full tablet report, sending...
I20260809 06:23:59.570904 23838 ts_manager.cc:194] Registered new tserver with Master: 1da870a7298e4e5aaf8bd5fb15efd369 (127.3.137.67:44199)
I20260809 06:23:59.571678 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.67:37057
I20260809 06:23:59.574564 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:23:59.584450 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.578829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.3.137.126:39107)
0809 06:23:59.579071 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.579076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.579562 (+ 486us) server_negotiation.cc:407] Connection header received
0809 06:23:59.579692 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.579696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.579764 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.579851 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:59.580320 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.581007 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.582275 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.582678 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.582765 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.583841 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:59.583845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:59.583848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:59.583864 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:59.583892 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:59.583903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:59.584005 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.584172 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.584215 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:23:59.588351 23838 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:23:59.595309 23838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:23:59.617991 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.607754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.3.137.67:44199)
0809 06:23:59.609085 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.609095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.609135 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:23:59.609241 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.609248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.609479 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.609689 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:59.610423 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.612102 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.615597 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.616717 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.616866 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.617158 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.617439 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.617620 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":997,"threads_started":1}
I20260809 06:23:59.618036 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.607245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:44199 (local address 127.0.0.1:53202)
0809 06:23:59.608224 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.608270 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.608422 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.609693 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.609700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.609735 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:59.610178 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.610191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.612242 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.612248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.615178 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.615193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.616903 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.616912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.617174 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.617239 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.617396 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.617599 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":153,"spinlock_wait_cycles":14208,"thread_start_us":805,"threads_started":1}
I20260809 06:23:59.619529 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.607405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49772)
0809 06:23:59.609731 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.609750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.609895 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.610756 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.610760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.610784 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:59.611277 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.611290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.612887 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.612892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.615179 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.615193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.617510 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.617517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.618749 (+ 1232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.618797 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.618921 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.619105 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"mutex_wait_us":70,"spinlock_wait_cycles":1280,"thread_start_us":2045,"threads_started":1}
I20260809 06:23:59.619547 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.607702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.3.137.65:44897)
0809 06:23:59.609340 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.609347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.609965 (+ 618us) server_negotiation.cc:407] Connection header received
0809 06:23:59.610074 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.610083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.610303 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.610513 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:59.611481 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.612753 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.615530 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.617014 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.617136 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.617378 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.619001 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.619176 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1367,"threads_started":1}
I20260809 06:23:59.621615 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.610525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:43331 (local address 127.0.0.1:36620)
0809 06:23:59.611423 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.611440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.611631 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.612509 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.612514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.612538 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:59.613021 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.613034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.615189 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.615194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.617293 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.617309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.620858 (+ 3549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.620865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.621027 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.621057 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.621154 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.621278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":1,"thread_start_us":610,"threads_started":1}
I20260809 06:23:59.621776 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.610177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.3.137.66:43331)
0809 06:23:59.610891 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.610902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.611536 (+ 634us) server_negotiation.cc:407] Connection header received
0809 06:23:59.611820 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.611827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.612052 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.612241 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:59.613267 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.615016 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.617516 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.618975 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.619112 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.619341 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.621262 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.621445 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":495,"threads_started":1}
I20260809 06:23:59.622643 24255 tablet_service.cc:1467] Processing CreateTablet for tablet f8559089033e403caa1954483c825aff (DEFAULT_TABLE table=test-table [id=419f254bf1bd4797887dda372ab57740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:59.623991 24255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8559089033e403caa1954483c825aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:59.624986 24111 tablet_service.cc:1467] Processing CreateTablet for tablet f8559089033e403caa1954483c825aff (DEFAULT_TABLE table=test-table [id=419f254bf1bd4797887dda372ab57740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:59.626031 24111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8559089033e403caa1954483c825aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:59.626940 23969 tablet_service.cc:1467] Processing CreateTablet for tablet f8559089033e403caa1954483c825aff (DEFAULT_TABLE table=test-table [id=419f254bf1bd4797887dda372ab57740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:23:59.628285 23969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8559089033e403caa1954483c825aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:23:59.637219 24346 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Bootstrap starting.
I20260809 06:23:59.641207 24346 tablet_bootstrap.cc:654] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:59.642863 24346 log.cc:824] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:59.650064 24347 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Bootstrap starting.
I20260809 06:23:59.650918 24349 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Bootstrap starting.
I20260809 06:23:59.653158 24346 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: No bootstrap required, opened a new log
I20260809 06:23:59.653311 24347 tablet_bootstrap.cc:654] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:59.653442 24346 ts_tablet_manager.cc:1397] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260809 06:23:59.654332 24347 log.cc:824] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:59.654433 24349 tablet_bootstrap.cc:654] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Neither blocks nor log segments found. Creating new log.
I20260809 06:23:59.655686 24349 log.cc:824] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:23:59.657258 24347 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: No bootstrap required, opened a new log
I20260809 06:23:59.657510 24347 ts_tablet_manager.cc:1397] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260809 06:23:59.658640 24349 tablet_bootstrap.cc:492] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: No bootstrap required, opened a new log
I20260809 06:23:59.658890 24349 ts_tablet_manager.cc:1397] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:23:59.660594 24041 heartbeater.cc:503] Master 127.3.137.126:39107 was elected leader, sending a full tablet report...
I20260809 06:23:59.663869 24346 raft_consensus.cc:348] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.664412 24346 raft_consensus.cc:374] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:59.664558 24346 raft_consensus.cc:729] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b492c5c83c90441e9511dce77c61bc81, State: Initialized, Role: FOLLOWER
I20260809 06:23:59.665336 24346 consensus_queue.cc:260] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.666842 24347 raft_consensus.cc:348] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.667371 24347 raft_consensus.cc:374] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:59.667522 24347 raft_consensus.cc:729] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da870a7298e4e5aaf8bd5fb15efd369, State: Initialized, Role: FOLLOWER
I20260809 06:23:59.667690 24184 heartbeater.cc:503] Master 127.3.137.126:39107 was elected leader, sending a full tablet report...
I20260809 06:23:59.668395 24346 ts_tablet_manager.cc:1428] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260809 06:23:59.668357 24347 consensus_queue.cc:260] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.669685 24349 raft_consensus.cc:348] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.670095 24349 raft_consensus.cc:374] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:23:59.670120 24347 ts_tablet_manager.cc:1428] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260809 06:23:59.670385 24325 heartbeater.cc:503] Master 127.3.137.126:39107 was elected leader, sending a full tablet report...
I20260809 06:23:59.671306 24349 raft_consensus.cc:729] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c951690e3d400a9d6a6489e6e3454f, State: Initialized, Role: FOLLOWER
I20260809 06:23:59.671983 24349 consensus_queue.cc:260] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.673774 24349 ts_tablet_manager.cc:1428] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260809 06:23:59.731655 24355 raft_consensus.cc:482] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:23:59.731989 24355 raft_consensus.cc:504] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.733776 24355 leader_election.cc:288] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da870a7298e4e5aaf8bd5fb15efd369 (127.3.137.67:44199), b492c5c83c90441e9511dce77c61bc81 (127.3.137.66:43331)
I20260809 06:23:59.741525 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.734183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:47345 (local address 127.3.137.67:44199)
0809 06:23:59.734407 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.734413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.735323 (+ 910us) server_negotiation.cc:407] Connection header received
0809 06:23:59.735595 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.735604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.735712 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.735808 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:59.736568 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.737530 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.739662 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.740560 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.740687 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.740736 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.741132 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.741263 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:23:59.741608 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.734484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:44769 (local address 127.3.137.66:43331)
0809 06:23:59.734920 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.734930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.736082 (+ 1152us) server_negotiation.cc:407] Connection header received
0809 06:23:59.736378 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.736385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.736477 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.736572 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:23:59.737142 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.737949 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.739929 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.740722 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.740882 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.740948 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.741225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.741352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260809 06:23:59.741658 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.734305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:44199 (local address 127.3.137.65:47345)
0809 06:23:59.735197 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.735213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.735394 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.735930 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.735935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.735961 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:59.736398 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.736411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.737661 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.737667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.739476 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.739489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.740751 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.740760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.740941 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.740988 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.741074 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.741186 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":82,"thread_start_us":1382,"threads_started":1}
I20260809 06:23:59.741737 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.734356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:43331 (local address 127.3.137.65:44769)
0809 06:23:59.735984 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.736002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:23:59.736219 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:23:59.736595 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:23:59.736600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:23:59.736631 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:23:59.736995 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.737007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.738103 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.738106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.739739 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:23:59.739754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.740904 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.740908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:23:59.741036 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.741089 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 06:23:59.741180 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:23:59.741309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"mutex_wait_us":25,"thread_start_us":458,"threads_started":1}
I20260809 06:23:59.742919 24134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8559089033e403caa1954483c825aff" candidate_uuid: "c6c951690e3d400a9d6a6489e6e3454f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b492c5c83c90441e9511dce77c61bc81" is_pre_election: true
I20260809 06:23:59.743000 24276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8559089033e403caa1954483c825aff" candidate_uuid: "c6c951690e3d400a9d6a6489e6e3454f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" is_pre_election: true
I20260809 06:23:59.743567 24276 raft_consensus.cc:2454] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c951690e3d400a9d6a6489e6e3454f in term 0.
I20260809 06:23:59.743567 24134 raft_consensus.cc:2454] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c951690e3d400a9d6a6489e6e3454f in term 0.
I20260809 06:23:59.744292 23922 leader_election.cc:302] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da870a7298e4e5aaf8bd5fb15efd369, c6c951690e3d400a9d6a6489e6e3454f; no voters:
I20260809 06:23:59.744706 24355 raft_consensus.cc:2781] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:23:59.744884 24355 raft_consensus.cc:482] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:23:59.745007 24355 raft_consensus.cc:3037] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:59.747277 24355 raft_consensus.cc:504] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.747843 24355 leader_election.cc:288] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [CANDIDATE]: Term 1 election: Requested vote from peers 1da870a7298e4e5aaf8bd5fb15efd369 (127.3.137.67:44199), b492c5c83c90441e9511dce77c61bc81 (127.3.137.66:43331)
I20260809 06:23:59.748304 24134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8559089033e403caa1954483c825aff" candidate_uuid: "c6c951690e3d400a9d6a6489e6e3454f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b492c5c83c90441e9511dce77c61bc81"
I20260809 06:23:59.748353 24276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8559089033e403caa1954483c825aff" candidate_uuid: "c6c951690e3d400a9d6a6489e6e3454f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da870a7298e4e5aaf8bd5fb15efd369"
I20260809 06:23:59.748487 24134 raft_consensus.cc:3037] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:59.748579 24276 raft_consensus.cc:3037] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:23:59.750751 24276 raft_consensus.cc:2454] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c951690e3d400a9d6a6489e6e3454f in term 1.
I20260809 06:23:59.750865 24134 raft_consensus.cc:2454] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c951690e3d400a9d6a6489e6e3454f in term 1.
I20260809 06:23:59.751165 23922 leader_election.cc:302] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da870a7298e4e5aaf8bd5fb15efd369, c6c951690e3d400a9d6a6489e6e3454f; no voters:
I20260809 06:23:59.751421 24355 raft_consensus.cc:2781] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:23:59.751849 24355 raft_consensus.cc:686] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 1 LEADER]: Becoming Leader. State: Replica: c6c951690e3d400a9d6a6489e6e3454f, State: Running, Role: LEADER
I20260809 06:23:59.752290 24355 consensus_queue.cc:237] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } }
I20260809 06:23:59.756915 23838 catalog_manager.cc:5495] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c951690e3d400a9d6a6489e6e3454f (127.3.137.65). New cstate: current_term: 1 leader_uuid: "c6c951690e3d400a9d6a6489e6e3454f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c951690e3d400a9d6a6489e6e3454f" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 44897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 } health_report { overall_health: UNKNOWN } } }
I20260809 06:23:59.777478 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.772029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.3.137.65:44897)
0809 06:23:59.772189 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.772193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.772311 (+ 118us) server_negotiation.cc:407] Connection header received
0809 06:23:59.772461 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.772464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.772542 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.772666 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:23:59.773010 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.773962 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.775437 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.775849 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.775951 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.776508 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:23:59.776963 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:23:59.777082 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.777254 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.777293 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:23:59.784854 24134 raft_consensus.cc:1261] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 1 FOLLOWER]: Refusing update from remote peer c6c951690e3d400a9d6a6489e6e3454f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:59.784914 24276 raft_consensus.cc:1261] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 1 FOLLOWER]: Refusing update from remote peer c6c951690e3d400a9d6a6489e6e3454f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:23:59.785794 24358 consensus_queue.cc:1035] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da870a7298e4e5aaf8bd5fb15efd369" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:59.786190 24355 consensus_queue.cc:1035] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b492c5c83c90441e9511dce77c61bc81" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:23:59.797111 24361 mvcc.cc:205] Tried to move back new op lower bound from 7316507196549607424 to 7316507196429156352. Current Snapshot: MvccSnapshot[applied={T|T < 7316507196549607424}]
I20260809 06:23:59.811075 23838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:59.811344 23838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:23:59.814874 23838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b86407f0af245cd96977703b9c613ba: Sending DeleteTablet for 3 replicas of tablet f8559089033e403caa1954483c825aff
I20260809 06:23:59.815969 24255 tablet_service.cc:1514] Processing DeleteTablet for tablet f8559089033e403caa1954483c825aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:59 UTC) from {username='slave'} at 127.0.0.1:53202
I20260809 06:23:59.816349 23969 tablet_service.cc:1514] Processing DeleteTablet for tablet f8559089033e403caa1954483c825aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:59 UTC) from {username='slave'} at 127.0.0.1:49772
I20260809 06:23:59.816416 24111 tablet_service.cc:1514] Processing DeleteTablet for tablet f8559089033e403caa1954483c825aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:23:59 UTC) from {username='slave'} at 127.0.0.1:36620
I20260809 06:23:59.818109 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23902
I20260809 06:23:59.828331 24371 tablet_replica.cc:330] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: stopping tablet replica
I20260809 06:23:59.828908 24371 raft_consensus.cc:2229] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:59.829509 24371 raft_consensus.cc:2258] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:59.831810 24371 ts_tablet_manager.cc:1905] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:59.839174 24371 ts_tablet_manager.cc:1918] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:59.839408 24371 log.cc:1196] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/f8559089033e403caa1954483c825aff
I20260809 06:23:59.840106 24371 ts_tablet_manager.cc:1939] T f8559089033e403caa1954483c825aff P 1da870a7298e4e5aaf8bd5fb15efd369: Deleting consensus metadata
I20260809 06:23:59.842072 23821 catalog_manager.cc:4841] TS 1da870a7298e4e5aaf8bd5fb15efd369 (127.3.137.67:44199): tablet f8559089033e403caa1954483c825aff (table test-table [id=419f254bf1bd4797887dda372ab57740]) successfully deleted
I20260809 06:23:59.845345 24372 tablet_replica.cc:330] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: stopping tablet replica
I20260809 06:23:59.846010 24372 raft_consensus.cc:2229] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:23:59.847026 24372 raft_consensus.cc:2258] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:59.847664 24373 tablet_replica.cc:330] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: stopping tablet replica
I20260809 06:23:59.848349 24373 raft_consensus.cc:2229] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:23:59.848963 24373 raft_consensus.cc:2258] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:23:59.849663 24372 ts_tablet_manager.cc:1905] T f8559089033e403caa1954483c825aff P c6c951690e3d400a9d6a6489e6e3454f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:59.851285 24373 ts_tablet_manager.cc:1905] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:23:59.853068 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.820295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.3.137.65:44897)
0809 06:23:59.821043 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.821053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.844292 (+ 23239us) server_negotiation.cc:407] Connection header received
0809 06:23:59.844529 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.844539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.844646 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.844766 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:59.845521 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.846803 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.848789 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.849418 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.849510 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.851268 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:59.851283 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:59.851300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:59.852104 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:59.852174 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:59.852202 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:59.852300 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.852566 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.852674 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260809 06:23:59.854339 24035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:23:59.858006 24373 ts_tablet_manager.cc:1918] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:23:59.858222 24373 log.cc:1196] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/f8559089033e403caa1954483c825aff
I20260809 06:23:59.858822 24373 ts_tablet_manager.cc:1939] T f8559089033e403caa1954483c825aff P b492c5c83c90441e9511dce77c61bc81: Deleting consensus metadata
I20260809 06:23:59.860706 23818 catalog_manager.cc:4841] TS b492c5c83c90441e9511dce77c61bc81 (127.3.137.66:43331): tablet f8559089033e403caa1954483c825aff (table test-table [id=419f254bf1bd4797887dda372ab57740]) successfully deleted
I20260809 06:23:59.952420 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23902
W20260809 06:23:59.974082 23818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:44897 (error 108)
I20260809 06:23:59.974409 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24046
I20260809 06:23:59.976301 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.975888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49794)
0809 06:23:59.976059 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 06:23:59.976167 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 06:23:59.976579 23818 catalog_manager.cc:4584] TS c6c951690e3d400a9d6a6489e6e3454f (127.3.137.65:44897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8559089033e403caa1954483c825aff: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
I20260809 06:23:59.982504 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:23:59.975346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.3.137.66:43331)
0809 06:23:59.975533 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:23:59.975538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:23:59.975689 (+ 151us) server_negotiation.cc:407] Connection header received
0809 06:23:59.975899 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:23:59.975905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:23:59.976051 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:23:59.976203 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:23:59.976713 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.977622 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.979126 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:23:59.979582 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:23:59.979737 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:23:59.980995 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:23:59.981004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:23:59.981018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:23:59.981627 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:23:59.981696 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:23:59.981724 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:23:59.981859 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 06:23:59.982022 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 06:23:59.982141 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:23:59.983335 24179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:00.030095 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.029594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49796)
0809 06:24:00.029833 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.029929 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 06:24:00.031286 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.030922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49800)
0809 06:24:00.031093 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.031166 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 06:24:00.073320 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24046
I20260809 06:24:00.091948 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.091489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49812)
0809 06:24:00.091697 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.091790 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:24:00.093029 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.092658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49820)
0809 06:24:00.092851 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.092920 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 06:24:00.096390 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24187
I20260809 06:24:00.103399 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.097131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.3.137.67:44199)
0809 06:24:00.097317 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:00.097325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:00.097348 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:24:00.097602 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:00.097608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:00.097714 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:00.097824 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:00.098270 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:00.099113 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:00.100425 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:00.100832 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:00.100966 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:00.102057 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:00.102067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:00.102082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:00.102652 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:00.102698 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:00.102718 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:00.102851 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:00.103004 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 06:24:00.103112 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:24:00.104084 24319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:00.173745 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.173259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49826)
0809 06:24:00.173484 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.173584 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 06:24:00.174867 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:00.174476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:44897 (local address 127.0.0.1:49832)
0809 06:24:00.174680 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:00.174761 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 06:24:00.195231 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24187
I20260809 06:24:00.215382 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 23800
I20260809 06:24:00.216563 23870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:00.325683 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 23800
2026-08-09T06:24:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2758 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:00 dist-test-slave-2w3w krb5kdc[24382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:00 dist-test-slave-2w3w krb5kdc[24382](info): set up 2 sockets
Aug 09 06:24:00 dist-test-slave-2w3w krb5kdc[24382](info): commencing operation
krb5kdc: starting...
W20260809 06:24:01.496490 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:01 dist-test-slave-2w3w krb5kdc[24382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:01.801973 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35393
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:02.139760 24395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:02.140034 24395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:02.140133 24395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:02.147181 24395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:02.147289 24395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:02.147336 24395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:02.147393 24395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:02.147439 24395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:02.156998 24395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:02.158222 24395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:02.159478 24395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:02.171645 24401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:02.171734 24402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:02.171921 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:02.172672 24395 server_base.cc:1034] running on GCE node
I20260809 06:24:02.173426 24395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:02.174887 24395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:02.176142 24395 hybrid_clock.cc:648] HybridClock initialized: now 1786256642176086 us; error 48 us; skew 500 ppm
I20260809 06:24:02.176494 24395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:02 dist-test-slave-2w3w krb5kdc[24382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:02.185544 24395 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:02.187266 24395 webserver.cc:460] Webserver started at http://127.3.137.126:39021/ using document root <none> and password file <none>
I20260809 06:24:02.188001 24395 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:02.188160 24395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:02.188500 24395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:02.191831 24395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "2dd2132407eb4b2d84ad5fa52fb98105"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "a83f090542e613c933046dcde9a2ad69"
server_key_iv: "415e77d8621cf2cae859824b85196ee0"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.192727 24395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "2dd2132407eb4b2d84ad5fa52fb98105"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "a83f090542e613c933046dcde9a2ad69"
server_key_iv: "415e77d8621cf2cae859824b85196ee0"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.198256 24395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:24:02.201905 24412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:02.203310 24395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:02.203505 24395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "2dd2132407eb4b2d84ad5fa52fb98105"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "a83f090542e613c933046dcde9a2ad69"
server_key_iv: "415e77d8621cf2cae859824b85196ee0"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.203722 24395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:02.214730 24395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:02.219161 24395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:02.219482 24395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:02.238051 24395 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:35393
I20260809 06:24:02.238075 24468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:35393 every 8 connection(s)
I20260809 06:24:02.239500 24395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:02.243182 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24395
I20260809 06:24:02.243222 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:02.243384 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:02.243881 3621 external_mini_cluster.cc:1428] Setting key 8215232f68cc39e3192e47e7c3888743
I20260809 06:24:02.257880 24469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: Bootstrap starting.
Aug 09 06:24:02 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:02.261814 24469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:02.263010 24469 log.cc:824] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:02.266314 24469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: No bootstrap required, opened a new log
I20260809 06:24:02.267576 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:02.245654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.3.137.126:35393)
0809 06:24:02.246493 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:02.246508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:02.246555 (+ 47us) server_negotiation.cc:407] Connection header received
0809 06:24:02.248407 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:02.248435 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:02.248937 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:02.249357 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:02.250816 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.252799 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.254104 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.255259 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.255462 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:02.260086 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:02.260094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:02.260108 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:02.260146 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:02.263427 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:02.264246 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:02.264255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:02.264265 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:02.264444 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:02.264926 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:02.264929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:02.264931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:02.265409 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:02.265627 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:02.266161 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 06:24:02.266553 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":330,"threads_started":1}
I20260809 06:24:02.274362 24469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } }
I20260809 06:24:02.274783 24469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:02.274895 24469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd2132407eb4b2d84ad5fa52fb98105, State: Initialized, Role: FOLLOWER
I20260809 06:24:02.275504 24469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } }
I20260809 06:24:02.275775 24469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:02.275949 24469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:02.276134 24469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:02.278644 24469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } }
I20260809 06:24:02.279235 24469 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd2132407eb4b2d84ad5fa52fb98105; no voters:
I20260809 06:24:02.279836 24469 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:02.279970 24475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:02.280710 24475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd2132407eb4b2d84ad5fa52fb98105, State: Running, Role: LEADER
I20260809 06:24:02.281308 24475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } }
I20260809 06:24:02.281839 24469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:02.283988 24477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd2132407eb4b2d84ad5fa52fb98105. Latest consensus state: current_term: 1 leader_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } } }
I20260809 06:24:02.284381 24477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:02.285018 24476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd2132407eb4b2d84ad5fa52fb98105" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35393 } } }
I20260809 06:24:02.285401 24476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:02.288198 24482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:02.294363 24482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:02.303618 24482 catalog_manager.cc:1347] Generated new cluster ID: d146d6edbeb24f9a8496cb5cbaaa4ba5
I20260809 06:24:02.303747 24482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:02.319288 24482 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:02.320520 24482 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:02.328562 24482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: Generated new TSK 0
I20260809 06:24:02.329497 24482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.137.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
Entry for principal kudu/127.3.137.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab.
I20260809 06:24:02.412482 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:02.751755 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:02.752010 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:02.752180 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:02.759280 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:02.759407 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:02.759588 24498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:24:02.769357 24498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.65.keytab
--principal=kudu/127.3.137.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:02.770617 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:02.771958 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:02.785513 24503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:02.785559 24504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:02.785494 24506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:02.786293 24498 server_base.cc:1034] running on GCE node
I20260809 06:24:02.786895 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:02.787997 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:02.789229 24498 hybrid_clock.cc:648] HybridClock initialized: now 1786256642789174 us; error 58 us; skew 500 ppm
I20260809 06:24:02.789551 24498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:02 dist-test-slave-2w3w krb5kdc[24382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:02.792850 24498 init.cc:377] Logged in from keytab as kudu/127.3.137.65@KRBTEST.COM (short username kudu)
I20260809 06:24:02.794204 24498 webserver.cc:460] Webserver started at http://127.3.137.65:33307/ using document root <none> and password file <none>
I20260809 06:24:02.794858 24498 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:02.795011 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:02.795374 24498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:02.798180 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "ce0c6980105c406e9b412d616b32b528"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "ce8dfc15ac1960cc23366ae3e9ce8576"
server_key_iv: "e28326a5940dbda44a037c4ede05e255"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.798933 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "ce0c6980105c406e9b412d616b32b528"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "ce8dfc15ac1960cc23366ae3e9ce8576"
server_key_iv: "e28326a5940dbda44a037c4ede05e255"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.803876 24498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:24:02.807178 24513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:02.808467 24498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:24:02.808635 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "ce0c6980105c406e9b412d616b32b528"
format_stamp: "Formatted at 2026-08-09 06:24:02 on dist-test-slave-2w3w"
server_key: "ce8dfc15ac1960cc23366ae3e9ce8576"
server_key_iv: "e28326a5940dbda44a037c4ede05e255"
server_key_version: "encryptionkey@0"
I20260809 06:24:02.808831 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:02.835175 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:02.839373 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:02.839602 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:02.840453 24498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:02.841626 24498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:02.841696 24498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:02.841785 24498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:02.841836 24498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:02.875236 24498 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:34367
I20260809 06:24:02.875283 24638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:34367 every 8 connection(s)
I20260809 06:24:02.876507 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:24:02.884523 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24498
I20260809 06:24:02.884699 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:24:02.885226 3621 external_mini_cluster.cc:1428] Setting key e4a7d63f86334ae6091c40c9c3e4af5c
Aug 09 06:24:02 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.65@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:02.896658 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:02.879131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:40325 (local address 127.3.137.126:35393)
0809 06:24:02.879597 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:02.879602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:02.880454 (+ 852us) server_negotiation.cc:407] Connection header received
0809 06:24:02.882258 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:02.882264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:02.882356 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:02.882481 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:02.884012 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.885151 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.886913 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.887374 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.887519 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:02.891097 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:02.891101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:02.891104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:02.891139 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:02.893665 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:02.894385 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:02.894395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:02.894397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:02.894516 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:02.894990 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:02.894995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:02.894999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:02.895259 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:02.895422 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:02.896085 (+ 663us) server_negotiation.cc:299] Negotiation successful
0809 06:24:02.896298 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260809 06:24:02.897589 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:02.879545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:35393 (local address 127.3.137.65:40325)
0809 06:24:02.880293 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:02.880329 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:02.881989 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:02.882717 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:02.882725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:02.883280 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:02.883827 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:02.883844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.885340 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.885347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:02.886741 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:02.886755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:02.887613 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:02.887623 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:02.887915 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:02.888614 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:02.888636 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:02.890857 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:02.893820 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:02.893829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:02.893843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:02.894247 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:02.894632 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:02.894637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:02.894641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:02.894843 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:02.895395 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:02.895403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:02.895716 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 06:24:02.895967 (+ 251us) client_negotiation.cc:240] Negotiation successful
0809 06:24:02.896470 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":353,"threads_started":1}
I20260809 06:24:02.899308 24639 heartbeater.cc:347] Connected to a master server at 127.3.137.126:35393
I20260809 06:24:02.899662 24639 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:02.900565 24639 heartbeater.cc:511] Master 127.3.137.126:35393 requested a full tablet report, sending...
I20260809 06:24:02.902885 24429 ts_manager.cc:194] Registered new tserver with Master: ce0c6980105c406e9b412d616b32b528 (127.3.137.65:34367)
I20260809 06:24:02.905078 24429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.65@KRBTEST.COM'} at 127.3.137.65:40325
WARNING: no policy specified for kudu/127.3.137.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
Entry for principal kudu/127.3.137.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab.
I20260809 06:24:02.954460 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:03.284201 24645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:03.284451 24645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:03.284543 24645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:03.291702 24645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:03.291803 24645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:03.291908 24645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:24:03.301380 24645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.66.keytab
--principal=kudu/127.3.137.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:03.302503 24645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:03.303740 24645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:03.316740 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:03.316808 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:03.316763 24655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:03.317625 24645 server_base.cc:1034] running on GCE node
I20260809 06:24:03.318288 24645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:03.319276 24645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:03.320508 24645 hybrid_clock.cc:648] HybridClock initialized: now 1786256643320471 us; error 46 us; skew 500 ppm
I20260809 06:24:03.320824 24645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:03 dist-test-slave-2w3w krb5kdc[24382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:03.323980 24645 init.cc:377] Logged in from keytab as kudu/127.3.137.66@KRBTEST.COM (short username kudu)
I20260809 06:24:03.325445 24645 webserver.cc:460] Webserver started at http://127.3.137.66:34391/ using document root <none> and password file <none>
I20260809 06:24:03.326110 24645 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:03.326260 24645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:03.326586 24645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:03.329679 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "98c2e6ac328b4596a62e26c49c868c4e"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "94aedb999554a3a5397734cf525468bd"
server_key_iv: "44065f1a4a7a85104cf4ff4a604bc8c8"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.330470 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "98c2e6ac328b4596a62e26c49c868c4e"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "94aedb999554a3a5397734cf525468bd"
server_key_iv: "44065f1a4a7a85104cf4ff4a604bc8c8"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.335532 24645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:24:03.338766 24664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.339994 24645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:03.340171 24645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "98c2e6ac328b4596a62e26c49c868c4e"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "94aedb999554a3a5397734cf525468bd"
server_key_iv: "44065f1a4a7a85104cf4ff4a604bc8c8"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.340370 24645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:03.366461 24645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:03.370419 24645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:03.370712 24645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:03.371677 24645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:03.372990 24645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:03.373100 24645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.373248 24645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:03.373323 24645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.407598 24645 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:45901
I20260809 06:24:03.407649 24789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:45901 every 8 connection(s)
I20260809 06:24:03.408928 24645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:24:03.412658 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24645
I20260809 06:24:03.412871 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:24:03.413511 3621 external_mini_cluster.cc:1428] Setting key be84f1b3bf7e898f135d1ee5787e4297
Aug 09 06:24:03 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.66@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:03.431480 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:03.411443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:55801 (local address 127.3.137.126:35393)
0809 06:24:03.411919 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:03.411925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:03.413189 (+ 1264us) server_negotiation.cc:407] Connection header received
0809 06:24:03.415581 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:03.415587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:03.415668 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:03.415778 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:03.417838 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.418558 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.420976 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.421377 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.421473 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:03.425610 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:03.425615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:03.425618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:03.425649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:03.428381 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.429118 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.429128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.429131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.429253 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.429712 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.429716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.429719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.429982 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:03.430144 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:03.430830 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 06:24:03.431107 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260809 06:24:03.432395 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:03.412054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:35393 (local address 127.3.137.66:55801)
0809 06:24:03.412993 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:03.413058 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:03.415262 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:03.416044 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:03.416057 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:03.416837 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:03.417660 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:03.417687 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.418711 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.418718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:03.420823 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:03.420846 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.421596 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.421606 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:03.421918 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:03.422658 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:03.422692 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:03.425353 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:03.428538 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:03.428547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:03.428562 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:03.428982 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:03.429371 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:03.429376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:03.429380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:03.429575 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:03.430125 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:03.430132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:03.430449 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 06:24:03.430721 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 06:24:03.431249 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":304,"threads_started":1}
I20260809 06:24:03.433862 24790 heartbeater.cc:347] Connected to a master server at 127.3.137.126:35393
I20260809 06:24:03.434186 24790 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:03.434885 24790 heartbeater.cc:511] Master 127.3.137.126:35393 requested a full tablet report, sending...
I20260809 06:24:03.436447 24429 ts_manager.cc:194] Registered new tserver with Master: 98c2e6ac328b4596a62e26c49c868c4e (127.3.137.66:45901)
I20260809 06:24:03.437726 24429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.66@KRBTEST.COM'} at 127.3.137.66:55801
WARNING: no policy specified for kudu/127.3.137.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
Entry for principal kudu/127.3.137.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab.
I20260809 06:24:03.480403 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:03.812845 24796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:03.813184 24796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:03.813315 24796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:03.820541 24796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:03.820642 24796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:03.820751 24796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:24:03.830263 24796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.67.keytab
--principal=kudu/127.3.137.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:03.831470 24796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:03.832732 24796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:03.845786 24802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:03.845736 24804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:03.845757 24801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:03.846720 24796 server_base.cc:1034] running on GCE node
I20260809 06:24:03.847383 24796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:03.848336 24796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:03.849562 24796 hybrid_clock.cc:648] HybridClock initialized: now 1786256643849508 us; error 68 us; skew 500 ppm
I20260809 06:24:03.849874 24796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:03 dist-test-slave-2w3w krb5kdc[24382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:03.853009 24796 init.cc:377] Logged in from keytab as kudu/127.3.137.67@KRBTEST.COM (short username kudu)
I20260809 06:24:03.854449 24796 webserver.cc:460] Webserver started at http://127.3.137.67:46337/ using document root <none> and password file <none>
I20260809 06:24:03.855193 24796 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:03.855343 24796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:03.855654 24796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:03.858544 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "8f505d95393243b1a039363815d3fb29"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "aed8f6f4fb12e4714b75cfb7a1548209"
server_key_iv: "8c4b9aa436e51c44e5324b97b8f41556"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.859378 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "8f505d95393243b1a039363815d3fb29"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "aed8f6f4fb12e4714b75cfb7a1548209"
server_key_iv: "8c4b9aa436e51c44e5324b97b8f41556"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.864426 24796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:24:03.867805 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.869112 24796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:24:03.869295 24796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "8f505d95393243b1a039363815d3fb29"
format_stamp: "Formatted at 2026-08-09 06:24:03 on dist-test-slave-2w3w"
server_key: "aed8f6f4fb12e4714b75cfb7a1548209"
server_key_iv: "8c4b9aa436e51c44e5324b97b8f41556"
server_key_version: "encryptionkey@0"
I20260809 06:24:03.869498 24796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:03.886217 24796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:03.890264 24796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:03.890564 24796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:03.891578 24796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:03.892877 24796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:03.892983 24796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.893158 24796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:03.893237 24796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:03.908044 24639 heartbeater.cc:503] Master 127.3.137.126:35393 was elected leader, sending a full tablet report...
I20260809 06:24:03.929431 24796 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:41667
I20260809 06:24:03.929481 24937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:41667 every 8 connection(s)
I20260809 06:24:03.930794 24796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:24:03.939436 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24796
I20260809 06:24:03.939615 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:24:03.940121 3621 external_mini_cluster.cc:1428] Setting key 84f2dcded138ce5b615fe59d8b7ea823
Aug 09 06:24:03 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.67@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:03.952842 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:03.933768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:45767 (local address 127.3.137.126:35393)
0809 06:24:03.934334 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:03.934339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:03.935313 (+ 974us) server_negotiation.cc:407] Connection header received
0809 06:24:03.937336 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:03.937343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:03.937440 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:03.937581 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:03.939310 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.940453 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.943027 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.943496 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.943592 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:03.947337 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:03.947342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:03.947345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:03.947375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:03.949926 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.950725 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.950728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.950731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.950872 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.951349 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.951352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.951354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.951603 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:03.951734 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:03.952403 (+ 669us) server_negotiation.cc:299] Negotiation successful
0809 06:24:03.952563 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":284,"threads_started":1}
I20260809 06:24:03.953946 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:03.934242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:35393 (local address 127.3.137.67:45767)
0809 06:24:03.935123 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:03.935175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:03.937017 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:03.937845 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:03.937854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:03.938419 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:03.939101 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:03.939123 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.940711 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.940718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:03.942837 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:03.942854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.943708 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.943713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:03.943941 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:03.944704 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:03.944728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:03.947160 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:03.950058 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:03.950065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:03.950078 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:03.950607 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:03.951027 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:03.951040 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:03.951043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:03.951224 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:03.951727 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:03.951733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:03.952072 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 06:24:03.952320 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 06:24:03.952698 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":296,"threads_started":1}
I20260809 06:24:03.955519 24938 heartbeater.cc:347] Connected to a master server at 127.3.137.126:35393
I20260809 06:24:03.955852 24938 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:03.956566 24938 heartbeater.cc:511] Master 127.3.137.126:35393 requested a full tablet report, sending...
I20260809 06:24:03.958212 24428 ts_manager.cc:194] Registered new tserver with Master: 8f505d95393243b1a039363815d3fb29 (127.3.137.67:41667)
I20260809 06:24:03.959304 24428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.137.67@KRBTEST.COM'} at 127.3.137.67:45767
I20260809 06:24:03.968381 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:24:03.986312 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:03.972892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.3.137.126:35393)
0809 06:24:03.973046 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:03.973051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:03.973547 (+ 496us) server_negotiation.cc:407] Connection header received
0809 06:24:03.973683 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:03.973687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:03.973758 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:03.973841 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:03.975647 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.976406 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.977841 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:03.978238 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:03.978336 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:03.980975 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:03.980980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:03.980983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:03.981015 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:03.983980 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.984621 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.984624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.984626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.984754 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:03.985073 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:03.985076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:03.985078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:03.985391 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:03.985567 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:03.985915 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 06:24:03.986076 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 06:24:03.990728 24428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:24:03.998001 24428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:24:04.022567 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.010475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:41667 (local address 127.0.0.1:37494)
0809 06:24:04.012907 (+ 2432us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:04.012950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:04.013128 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:04.013927 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:04.013935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:04.013967 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:04.014469 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.014483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.016906 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.016912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.019547 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.019567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.021431 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.021438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.021670 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.021749 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 06:24:04.021876 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 06:24:04.022062 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2138,"mutex_wait_us":95,"thread_start_us":2946,"threads_started":1}
I20260809 06:24:04.022614 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.010652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:45901 (local address 127.0.0.1:53166)
0809 06:24:04.013684 (+ 3032us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:04.013707 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:04.013825 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:04.015740 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:04.015744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:04.015788 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:04.016179 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.016191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.018428 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.018433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.020317 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.020329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.021756 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.021762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.021959 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.021990 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:24:04.022048 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 06:24:04.022147 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2918,"mutex_wait_us":1,"spinlock_wait_cycles":16640,"thread_start_us":1831,"threads_started":1}
I20260809 06:24:04.023540 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.011265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.3.137.66:45901)
0809 06:24:04.012915 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.012928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.013950 (+ 1022us) server_negotiation.cc:407] Connection header received
0809 06:24:04.015224 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.015262 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.015520 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.016359 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:04.016605 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.018259 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.020486 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.021518 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.022704 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.022904 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.022999 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.023201 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":1785,"threads_started":1}
I20260809 06:24:04.023973 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.015025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:34367 (local address 127.0.0.1:52066)
0809 06:24:04.015637 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:04.015666 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:04.015803 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:04.016874 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:04.016879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:04.016903 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:04.017382 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.017394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.019175 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.019187 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.021212 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.021225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.023319 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.023325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.023457 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.023480 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:24:04.023530 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 06:24:04.023646 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":33,"spinlock_wait_cycles":9728,"thread_start_us":461,"threads_started":1}
I20260809 06:24:04.024032 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.010794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.3.137.67:41667)
0809 06:24:04.013157 (+ 2363us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.013168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.013211 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:24:04.013335 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.013346 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.013619 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.014695 (+ 1076us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:04.015049 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.016724 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.019738 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.021216 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.021366 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.021616 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.023511 (+ 1895us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.023697 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1524,"thread_start_us":1315,"threads_started":1}
I20260809 06:24:04.025053 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.014409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.3.137.65:34367)
0809 06:24:04.016433 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.016439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.016461 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:24:04.016531 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.016536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.016690 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.016887 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:04.017638 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.019026 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.021434 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.023057 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.024214 (+ 1157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.024481 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.024560 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.024725 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1879,"thread_start_us":1699,"threads_started":1}
I20260809 06:24:04.027577 24866 tablet_service.cc:1467] Processing CreateTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e (DEFAULT_TABLE table=test-table [id=15851c7963e64bde966643d7b49a8d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:04.027515 24716 tablet_service.cc:1467] Processing CreateTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e (DEFAULT_TABLE table=test-table [id=15851c7963e64bde966643d7b49a8d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:04.028828 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89210c6dd0ea44e1baab498cffd4ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:04.028954 24716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89210c6dd0ea44e1baab498cffd4ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:04.028503 24565 tablet_service.cc:1467] Processing CreateTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e (DEFAULT_TABLE table=test-table [id=15851c7963e64bde966643d7b49a8d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:04.029762 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89210c6dd0ea44e1baab498cffd4ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:04.042069 24957 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Bootstrap starting.
I20260809 06:24:04.044107 24959 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Bootstrap starting.
I20260809 06:24:04.045818 24958 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Bootstrap starting.
I20260809 06:24:04.046000 24957 tablet_bootstrap.cc:654] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:04.047340 24957 log.cc:824] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:04.048316 24959 tablet_bootstrap.cc:654] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:04.048763 24958 tablet_bootstrap.cc:654] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:04.049532 24959 log.cc:824] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:04.049961 24958 log.cc:824] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:04.050851 24957 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: No bootstrap required, opened a new log
I20260809 06:24:04.051131 24957 ts_tablet_manager.cc:1397] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 06:24:04.052578 24959 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: No bootstrap required, opened a new log
I20260809 06:24:04.052847 24959 ts_tablet_manager.cc:1397] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:24:04.052978 24958 tablet_bootstrap.cc:492] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: No bootstrap required, opened a new log
I20260809 06:24:04.053224 24958 ts_tablet_manager.cc:1397] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 06:24:04.059773 24959 raft_consensus.cc:348] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.060242 24959 raft_consensus.cc:374] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:04.060376 24959 raft_consensus.cc:729] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c6980105c406e9b412d616b32b528, State: Initialized, Role: FOLLOWER
I20260809 06:24:04.060963 24959 consensus_queue.cc:260] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.061872 24957 raft_consensus.cc:348] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.062338 24957 raft_consensus.cc:374] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:04.062489 24957 raft_consensus.cc:729] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f505d95393243b1a039363815d3fb29, State: Initialized, Role: FOLLOWER
I20260809 06:24:04.062696 24959 ts_tablet_manager.cc:1428] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260809 06:24:04.063246 24957 consensus_queue.cc:260] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.063936 24958 raft_consensus.cc:348] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.064306 24938 heartbeater.cc:503] Master 127.3.137.126:35393 was elected leader, sending a full tablet report...
I20260809 06:24:04.064597 24958 raft_consensus.cc:374] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:04.064702 24958 raft_consensus.cc:729] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c2e6ac328b4596a62e26c49c868c4e, State: Initialized, Role: FOLLOWER
I20260809 06:24:04.064946 24957 ts_tablet_manager.cc:1428] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260809 06:24:04.065454 24958 consensus_queue.cc:260] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.066623 24790 heartbeater.cc:503] Master 127.3.137.126:35393 was elected leader, sending a full tablet report...
I20260809 06:24:04.067272 24958 ts_tablet_manager.cc:1428] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260809 06:24:04.129012 24640 tablet.cc:2369] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:24:04.161091 24791 tablet.cc:2369] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:24:04.182578 24939 tablet.cc:2369] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:04.408107 24964 raft_consensus.cc:482] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:24:04.408510 24964 raft_consensus.cc:504] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.410511 24964 leader_election.cc:288] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f505d95393243b1a039363815d3fb29 (127.3.137.67:41667), 98c2e6ac328b4596a62e26c49c868c4e (127.3.137.66:45901)
I20260809 06:24:04.419955 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.411496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:45191 (local address 127.3.137.67:41667)
0809 06:24:04.411750 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.411760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.413330 (+ 1570us) server_negotiation.cc:407] Connection header received
0809 06:24:04.413573 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.413582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.413660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.413774 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:04.414322 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.415459 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.417459 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.418871 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.419090 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.419177 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.419539 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.419673 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:24:04.420248 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.411208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:55057 (local address 127.3.137.66:45901)
0809 06:24:04.411488 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.411499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.412464 (+ 965us) server_negotiation.cc:407] Connection header received
0809 06:24:04.412751 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.412758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.412858 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.412970 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:04.413792 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.415129 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.417677 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.419205 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.419380 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.419476 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.419798 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.419937 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 06:24:04.420352 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.410892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:45901 (local address 127.3.137.65:55057)
0809 06:24:04.412339 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:04.412361 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:04.412586 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:04.413132 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:04.413138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:04.413176 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:04.413620 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.413631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.415295 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.415301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.417512 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.417528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.419451 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.419459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.419626 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.419651 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:24:04.419768 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 06:24:04.419910 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":1324,"threads_started":1}
I20260809 06:24:04.420753 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.410946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:41667 (local address 127.3.137.65:45191)
0809 06:24:04.413215 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:04.413235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:04.413400 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:04.413794 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:04.413798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:04.413819 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:04.414144 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.414153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.415612 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.415619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.417268 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:04.417282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.419145 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.419154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:04.419341 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.419380 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 06:24:04.420229 (+ 849us) client_negotiation.cc:240] Negotiation successful
0809 06:24:04.420388 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"thread_start_us":1762,"threads_started":1}
I20260809 06:24:04.421643 24740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89210c6dd0ea44e1baab498cffd4ec4e" candidate_uuid: "ce0c6980105c406e9b412d616b32b528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c2e6ac328b4596a62e26c49c868c4e" is_pre_election: true
I20260809 06:24:04.421640 24889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89210c6dd0ea44e1baab498cffd4ec4e" candidate_uuid: "ce0c6980105c406e9b412d616b32b528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f505d95393243b1a039363815d3fb29" is_pre_election: true
I20260809 06:24:04.422215 24889 raft_consensus.cc:2454] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0c6980105c406e9b412d616b32b528 in term 0.
I20260809 06:24:04.422219 24740 raft_consensus.cc:2454] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0c6980105c406e9b412d616b32b528 in term 0.
I20260809 06:24:04.423056 24516 leader_election.cc:302] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98c2e6ac328b4596a62e26c49c868c4e, ce0c6980105c406e9b412d616b32b528; no voters:
I20260809 06:24:04.423601 24964 raft_consensus.cc:2781] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:24:04.423822 24964 raft_consensus.cc:482] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:24:04.423949 24964 raft_consensus.cc:3037] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:04.426254 24964 raft_consensus.cc:504] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.426915 24964 leader_election.cc:288] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [CANDIDATE]: Term 1 election: Requested vote from peers 8f505d95393243b1a039363815d3fb29 (127.3.137.67:41667), 98c2e6ac328b4596a62e26c49c868c4e (127.3.137.66:45901)
I20260809 06:24:04.427444 24889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89210c6dd0ea44e1baab498cffd4ec4e" candidate_uuid: "ce0c6980105c406e9b412d616b32b528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f505d95393243b1a039363815d3fb29"
I20260809 06:24:04.427444 24740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89210c6dd0ea44e1baab498cffd4ec4e" candidate_uuid: "ce0c6980105c406e9b412d616b32b528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c2e6ac328b4596a62e26c49c868c4e"
I20260809 06:24:04.427680 24889 raft_consensus.cc:3037] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:04.427678 24740 raft_consensus.cc:3037] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:04.431367 24740 raft_consensus.cc:2454] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0c6980105c406e9b412d616b32b528 in term 1.
I20260809 06:24:04.431367 24889 raft_consensus.cc:2454] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0c6980105c406e9b412d616b32b528 in term 1.
I20260809 06:24:04.431818 24516 leader_election.cc:302] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98c2e6ac328b4596a62e26c49c868c4e, ce0c6980105c406e9b412d616b32b528; no voters:
I20260809 06:24:04.432142 24964 raft_consensus.cc:2781] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:04.432605 24964 raft_consensus.cc:686] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 1 LEADER]: Becoming Leader. State: Replica: ce0c6980105c406e9b412d616b32b528, State: Running, Role: LEADER
I20260809 06:24:04.433228 24964 consensus_queue.cc:237] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } }
I20260809 06:24:04.438308 24429 catalog_manager.cc:5495] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce0c6980105c406e9b412d616b32b528 (127.3.137.65). New cstate: current_term: 1 leader_uuid: "ce0c6980105c406e9b412d616b32b528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce0c6980105c406e9b412d616b32b528" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 34367 } health_report { overall_health: HEALTHY } } }
I20260809 06:24:04.475160 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.469411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.3.137.65:34367)
0809 06:24:04.469665 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.469670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.469688 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:24:04.469932 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.469936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.470023 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.470144 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:04.470503 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.471369 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.472826 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.473325 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.473420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.473904 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:04.474489 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:04.474609 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.474756 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.474824 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 06:24:04.483251 24740 raft_consensus.cc:1261] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 1 FOLLOWER]: Refusing update from remote peer ce0c6980105c406e9b412d616b32b528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:04.483394 24889 raft_consensus.cc:1261] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 1 FOLLOWER]: Refusing update from remote peer ce0c6980105c406e9b412d616b32b528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:04.484282 24972 consensus_queue.cc:1035] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c2e6ac328b4596a62e26c49c868c4e" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:04.484622 24964 consensus_queue.cc:1035] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f505d95393243b1a039363815d3fb29" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:04.503098 24976 mvcc.cc:205] Tried to move back new op lower bound from 7316507215793475584 to 7316507215603916800. Current Snapshot: MvccSnapshot[applied={T|T < 7316507215793475584}]
I20260809 06:24:04.516505 24427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:24:04.516889 24427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:24:04.520515 24427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd2132407eb4b2d84ad5fa52fb98105: Sending DeleteTablet for 3 replicas of tablet 89210c6dd0ea44e1baab498cffd4ec4e
I20260809 06:24:04.521840 24866 tablet_service.cc:1514] Processing DeleteTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:04 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:37494
I20260809 06:24:04.521853 24716 tablet_service.cc:1514] Processing DeleteTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:04 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:53166
I20260809 06:24:04.521858 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet 89210c6dd0ea44e1baab498cffd4ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:04 UTC) from {username='kudu', principal='kudu/127.3.137.126@KRBTEST.COM'} at 127.0.0.1:52066
I20260809 06:24:04.524545 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24498
I20260809 06:24:04.534724 24987 tablet_replica.cc:330] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: stopping tablet replica
I20260809 06:24:04.535485 24985 tablet_replica.cc:330] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: stopping tablet replica
I20260809 06:24:04.536293 24987 raft_consensus.cc:2229] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:24:04.536934 24985 raft_consensus.cc:2229] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:24:04.537678 24985 raft_consensus.cc:2258] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:24:04.538523 24987 raft_consensus.cc:2258] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:24:04.541718 24985 ts_tablet_manager.cc:1905] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:04.543825 24986 tablet_replica.cc:330] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: stopping tablet replica
I20260809 06:24:04.544584 24987 ts_tablet_manager.cc:1905] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:04.545863 24986 raft_consensus.cc:2229] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:24:04.546367 24986 raft_consensus.cc:2258] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 09 06:24:04 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.65@KRBTEST.COM
I20260809 06:24:04.552320 24986 ts_tablet_manager.cc:1905] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:04.552531 24987 ts_tablet_manager.cc:1918] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:24:04.552815 24987 log.cc:1196] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-0/wal/wals/89210c6dd0ea44e1baab498cffd4ec4e
I20260809 06:24:04.553534 24987 ts_tablet_manager.cc:1939] T 89210c6dd0ea44e1baab498cffd4ec4e P ce0c6980105c406e9b412d616b32b528: Deleting consensus metadata
I20260809 06:24:04.555620 24416 catalog_manager.cc:4841] TS ce0c6980105c406e9b412d616b32b528 (127.3.137.65:34367): tablet 89210c6dd0ea44e1baab498cffd4ec4e (table test-table [id=15851c7963e64bde966643d7b49a8d4b]) successfully deleted
I20260809 06:24:04.558998 24985 ts_tablet_manager.cc:1918] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:24:04.559244 24985 log.cc:1196] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/89210c6dd0ea44e1baab498cffd4ec4e
I20260809 06:24:04.559864 24985 ts_tablet_manager.cc:1939] T 89210c6dd0ea44e1baab498cffd4ec4e P 8f505d95393243b1a039363815d3fb29: Deleting consensus metadata
I20260809 06:24:04.560667 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.525849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.3.137.65:34367)
0809 06:24:04.526079 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.526089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.539604 (+ 13515us) server_negotiation.cc:407] Connection header received
0809 06:24:04.540060 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.540067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.540156 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.540239 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:04.542141 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.543399 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.545457 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.546234 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.546380 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.553234 (+ 6854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:04.553250 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:04.553269 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:04.553321 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:04.557081 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.557798 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.557806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.557809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.557993 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.558466 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.558471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.558474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.559041 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:04.559337 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.560003 (+ 666us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.560261 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:24:04.561663 24416 catalog_manager.cc:4841] TS 8f505d95393243b1a039363815d3fb29 (127.3.137.67:41667): tablet 89210c6dd0ea44e1baab498cffd4ec4e (table test-table [id=15851c7963e64bde966643d7b49a8d4b]) successfully deleted
I20260809 06:24:04.562304 24986 ts_tablet_manager.cc:1918] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:24:04.562469 24986 log.cc:1196] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/89210c6dd0ea44e1baab498cffd4ec4e
I20260809 06:24:04.562736 24633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:04.563002 24986 ts_tablet_manager.cc:1939] T 89210c6dd0ea44e1baab498cffd4ec4e P 98c2e6ac328b4596a62e26c49c868c4e: Deleting consensus metadata
I20260809 06:24:04.564826 24415 catalog_manager.cc:4841] TS 98c2e6ac328b4596a62e26c49c868c4e (127.3.137.66:45901): tablet 89210c6dd0ea44e1baab498cffd4ec4e (table test-table [id=15851c7963e64bde966643d7b49a8d4b]) successfully deleted
I20260809 06:24:04.662318 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24498
I20260809 06:24:04.684022 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24645
Aug 09 06:24:04 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.66@KRBTEST.COM
I20260809 06:24:04.699533 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.684803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.3.137.66:45901)
0809 06:24:04.685018 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.685024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.685044 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:24:04.685203 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.685209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.685292 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.685400 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:04.686666 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.687655 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.689157 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.689589 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.689730 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.694306 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:04.694320 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:04.694335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:04.694368 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:04.697079 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.697711 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.697718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.697720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.697852 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.698174 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.698177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.698179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.698545 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:04.698750 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.699091 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.699253 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:24:04.700248 24784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:04.791282 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24645
I20260809 06:24:04.810551 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24796
Aug 09 06:24:04 dist-test-slave-2w3w krb5kdc[24382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.67@KRBTEST.COM
I20260809 06:24:04.825547 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:04.811447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.3.137.67:41667)
0809 06:24:04.811616 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:04.811621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:04.811640 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:24:04.811788 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:04.811792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:04.811869 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:04.811959 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:04.813284 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.814006 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.815457 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:04.815874 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:04.815973 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:04.820293 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:04.820308 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:04.820324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:04.820357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:04.823080 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.823736 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.823743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.823745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.823916 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:04.824222 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:04.824224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:04.824226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:04.824564 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:04.824801 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:04.825007 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 06:24:04.825163 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:24:04.826226 24932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:04.915230 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24796
I20260809 06:24:04.935051 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24395
I20260809 06:24:04.936210 24464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:05.052457 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24395
2026-08-09T06:24:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4726 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T06:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:05Z Disabled control of system clock
I20260809 06:24:05.145624 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42153
--webserver_interface=127.3.137.126
--webserver_port=0
--builtin_ntp_servers=127.3.137.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:24:05.483124 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:05.483413 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:05.483558 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:05.490859 24997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:05.491039 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:05.491129 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:05.491245 24997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:05.491297 24997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:05.500686 24997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:42153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:05.501986 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:05.503880 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:05.515654 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:05.515698 25007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:05.515736 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:05.516641 24997 server_base.cc:1034] running on GCE node
I20260809 06:24:05.517464 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:05.518863 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:05.520169 24997 hybrid_clock.cc:648] HybridClock initialized: now 1786256645520126 us; error 35 us; skew 500 ppm
I20260809 06:24:05.520562 24997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:24:05.522877 24997 webserver.cc:460] Webserver started at http://127.3.137.126:39515/ using document root <none> and password file <none>
I20260809 06:24:05.523600 24997 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:05.523736 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:05.524077 24997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:05.527037 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "a3bdafd1975e48ddb4e9f92454a39c46"
format_stamp: "Formatted at 2026-08-09 06:24:05 on dist-test-slave-2w3w"
server_key: "67ac634c8db4151b50ce6d241a69f730"
server_key_iv: "76374ac73f6047febc89c1e272801732"
server_key_version: "encryptionkey@0"
I20260809 06:24:05.527879 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "a3bdafd1975e48ddb4e9f92454a39c46"
format_stamp: "Formatted at 2026-08-09 06:24:05 on dist-test-slave-2w3w"
server_key: "67ac634c8db4151b50ce6d241a69f730"
server_key_iv: "76374ac73f6047febc89c1e272801732"
server_key_version: "encryptionkey@0"
I20260809 06:24:05.532943 24997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:24:05.536382 25013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:05.537725 24997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 06:24:05.537925 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "a3bdafd1975e48ddb4e9f92454a39c46"
format_stamp: "Formatted at 2026-08-09 06:24:05 on dist-test-slave-2w3w"
server_key: "67ac634c8db4151b50ce6d241a69f730"
server_key_iv: "76374ac73f6047febc89c1e272801732"
server_key_version: "encryptionkey@0"
I20260809 06:24:05.538142 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:05.588572 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:05.589511 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:05.589840 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:05.608536 24997 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:42153
I20260809 06:24:05.608575 25068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:42153 every 8 connection(s)
I20260809 06:24:05.609833 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:05.613443 25069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:05.614790 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 24997
I20260809 06:24:05.615031 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:05.615497 3621 external_mini_cluster.cc:1428] Setting key 4d864966a79e3f317ae4470e3043dd1a
I20260809 06:24:05.627920 25069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: Bootstrap starting.
I20260809 06:24:05.630324 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:05.617454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.3.137.126:42153)
0809 06:24:05.618554 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:05.618572 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:05.618631 (+ 59us) server_negotiation.cc:407] Connection header received
0809 06:24:05.620277 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:05.620307 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:05.620895 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:05.621319 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:05.621812 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:05.623747 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:05.625086 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:05.626122 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:05.626296 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:05.627516 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:05.627528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:05.627545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:05.628230 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:05.628352 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:05.628380 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:05.628531 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:05.628737 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 06:24:05.629037 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260809 06:24:05.631794 25069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:05.633131 25069 log.cc:824] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:05.636240 25069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: No bootstrap required, opened a new log
I20260809 06:24:05.644338 25069 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } }
I20260809 06:24:05.644702 25069 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:05.644806 25069 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bdafd1975e48ddb4e9f92454a39c46, State: Initialized, Role: FOLLOWER
I20260809 06:24:05.645351 25069 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } }
I20260809 06:24:05.645603 25069 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:05.645737 25069 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:05.645908 25069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:05.648142 25069 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } }
I20260809 06:24:05.648638 25069 leader_election.cc:302] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3bdafd1975e48ddb4e9f92454a39c46; no voters:
I20260809 06:24:05.649232 25069 leader_election.cc:288] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:05.649398 25074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:05.651098 25074 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [term 1 LEADER]: Becoming Leader. State: Replica: a3bdafd1975e48ddb4e9f92454a39c46, State: Running, Role: LEADER
I20260809 06:24:05.651361 25069 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:05.651671 25074 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } }
I20260809 06:24:05.654953 25075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } } }
I20260809 06:24:05.655490 25075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:05.654944 25076 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3bdafd1975e48ddb4e9f92454a39c46. Latest consensus state: current_term: 1 leader_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bdafd1975e48ddb4e9f92454a39c46" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 42153 } } }
I20260809 06:24:05.656011 25076 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:05.657013 25082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:05.664716 25082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:05.674762 25082 catalog_manager.cc:1347] Generated new cluster ID: 9331f12fb5d84ef9b7e07c4f1ae17be4
I20260809 06:24:05.674901 25082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:05.696641 25082 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:05.698570 25082 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:05.711336 25082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: Generated new TSK 0
I20260809 06:24:05.712198 25082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:05.722581 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.65:0
--local_ip_for_outbound_sockets=127.3.137.65
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--builtin_ntp_servers=127.3.137.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:24:06.065361 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:06.065649 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:06.065822 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:06.073058 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:06.073153 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:06.073341 25094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.65
I20260809 06:24:06.082387 25094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.137.65
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.65
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:06.083784 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:06.085626 25094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:06.099141 25101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:06.099092 25104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:06.099184 25102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:06.100272 25094 server_base.cc:1034] running on GCE node
I20260809 06:24:06.100875 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:06.101744 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:06.103001 25094 hybrid_clock.cc:648] HybridClock initialized: now 1786256646102912 us; error 42 us; skew 500 ppm
I20260809 06:24:06.103299 25094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:24:06.105527 25094 webserver.cc:460] Webserver started at http://127.3.137.65:42597/ using document root <none> and password file <none>
I20260809 06:24:06.106221 25094 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:06.106316 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:06.106590 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:06.109495 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data/instance:
uuid: "2115d85d17664e3fb01425ab0380007c"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "04073720563e93f5aedbded257ab3b57"
server_key_iv: "24c6abb3e9ca89fb04b88f07de5ad720"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.110268 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance:
uuid: "2115d85d17664e3fb01425ab0380007c"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "04073720563e93f5aedbded257ab3b57"
server_key_iv: "24c6abb3e9ca89fb04b88f07de5ad720"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.115439 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:24:06.118706 25110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.120064 25094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:06.120198 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal
uuid: "2115d85d17664e3fb01425ab0380007c"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "04073720563e93f5aedbded257ab3b57"
server_key_iv: "24c6abb3e9ca89fb04b88f07de5ad720"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.120373 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:06.141095 25094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:06.141863 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:06.142125 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:06.143177 25094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:06.144601 25094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:06.144692 25094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.144783 25094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:06.144838 25094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.177834 25094 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.65:43419
I20260809 06:24:06.177939 25233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.65:43419 every 8 connection(s)
I20260809 06:24:06.179210 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/data/info.pb
I20260809 06:24:06.187736 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25094
I20260809 06:24:06.187922 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-0/wal/instance
I20260809 06:24:06.188505 3621 external_mini_cluster.cc:1428] Setting key 2e2d1d0a7c14b9df84f1f4f87d81117d
I20260809 06:24:06.192051 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.66:0
--local_ip_for_outbound_sockets=127.3.137.66
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--builtin_ntp_servers=127.3.137.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:24:06.193856 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:06.181774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:55497 (local address 127.3.137.126:42153)
0809 06:24:06.182211 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:06.182216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:06.183448 (+ 1232us) server_negotiation.cc:407] Connection header received
0809 06:24:06.185266 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:06.185270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:06.185337 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:06.185434 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:06.186644 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.187693 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.190548 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.191324 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.191465 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:06.192900 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:06.192907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:06.192911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:06.192932 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:06.192982 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:06.192998 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:06.193108 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:06.193401 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 06:24:06.193505 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260809 06:24:06.195338 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:06.182356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42153 (local address 127.3.137.65:55497)
0809 06:24:06.183292 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:06.183334 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:06.185028 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:06.185625 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:06.185633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:06.185711 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:06.186441 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:06.186470 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.187960 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.187966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:06.190368 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:06.190401 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.191607 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.191616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:06.191854 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:06.192575 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:24:06.192578 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:06.192620 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:24:06.192703 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:06.193124 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:06.193136 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:06.193161 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:24:06.193374 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 06:24:06.193769 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260809 06:24:06.197796 25234 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42153
I20260809 06:24:06.198247 25234 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:06.199527 25234 heartbeater.cc:511] Master 127.3.137.126:42153 requested a full tablet report, sending...
I20260809 06:24:06.202265 25032 ts_manager.cc:194] Registered new tserver with Master: 2115d85d17664e3fb01425ab0380007c (127.3.137.65:43419)
I20260809 06:24:06.204543 25032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.65:55497
W20260809 06:24:06.539069 25239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:06.539379 25239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:06.539544 25239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:06.546772 25239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:06.546903 25239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:06.547093 25239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.66
I20260809 06:24:06.556191 25239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.137.66
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.66
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:06.557533 25239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:06.559485 25239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:06.573096 25246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:06.573061 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:06.573133 25247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:06.573788 25239 server_base.cc:1034] running on GCE node
I20260809 06:24:06.574376 25239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:06.575397 25239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:06.576634 25239 hybrid_clock.cc:648] HybridClock initialized: now 1786256646576592 us; error 54 us; skew 500 ppm
I20260809 06:24:06.576947 25239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:24:06.579231 25239 webserver.cc:460] Webserver started at http://127.3.137.66:40855/ using document root <none> and password file <none>
I20260809 06:24:06.579903 25239 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:06.580044 25239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:06.580368 25239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:06.583179 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data/instance:
uuid: "0bed09c4520749cd8df02207be7afd93"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "83b7c1e1da63735e2ab8f584d1e43e67"
server_key_iv: "192e809813f9e1a48b5c859b043502ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.583994 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance:
uuid: "0bed09c4520749cd8df02207be7afd93"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "83b7c1e1da63735e2ab8f584d1e43e67"
server_key_iv: "192e809813f9e1a48b5c859b043502ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.589054 25239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:24:06.592407 25255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.593636 25239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:24:06.593814 25239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal
uuid: "0bed09c4520749cd8df02207be7afd93"
format_stamp: "Formatted at 2026-08-09 06:24:06 on dist-test-slave-2w3w"
server_key: "83b7c1e1da63735e2ab8f584d1e43e67"
server_key_iv: "192e809813f9e1a48b5c859b043502ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:06.594074 25239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:06.613747 25239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:06.614723 25239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:06.615070 25239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:06.616190 25239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:06.617937 25239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:06.618062 25239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.618198 25239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:06.618270 25239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:06.653383 25239 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.66:41127
I20260809 06:24:06.653474 25379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.66:41127 every 8 connection(s)
I20260809 06:24:06.654758 25239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/data/info.pb
I20260809 06:24:06.661194 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25239
I20260809 06:24:06.661365 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal/instance
I20260809 06:24:06.661831 3621 external_mini_cluster.cc:1428] Setting key a99debcbf04959740092dfaefbce144d
I20260809 06:24:06.664712 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.137.67:0
--local_ip_for_outbound_sockets=127.3.137.67
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--builtin_ntp_servers=127.3.137.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:24:06.667471 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:06.657182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.66:33667 (local address 127.3.137.126:42153)
0809 06:24:06.657404 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:06.657411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:06.658891 (+ 1480us) server_negotiation.cc:407] Connection header received
0809 06:24:06.660620 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:06.660624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:06.660690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:06.660779 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:06.661891 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.662779 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.664465 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.665152 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.665298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:06.666559 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:06.666568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:06.666576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:06.666601 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:06.666642 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:06.666661 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:06.666757 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:06.666988 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 06:24:06.667099 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:24:06.668300 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:06.657734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42153 (local address 127.3.137.66:33667)
0809 06:24:06.658622 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:06.658657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:06.660384 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:06.660997 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:06.661004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:06.661083 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:06.661715 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:06.661732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.662948 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.662952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:06.664315 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:06.664327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:06.665354 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:06.665358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:06.665549 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:06.666239 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:24:06.666242 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:06.666279 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:24:06.666355 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:06.666760 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:06.666768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:06.666773 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:24:06.666908 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 06:24:06.667248 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260809 06:24:06.669679 25380 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42153
I20260809 06:24:06.669972 25380 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:06.670678 25380 heartbeater.cc:511] Master 127.3.137.126:42153 requested a full tablet report, sending...
I20260809 06:24:06.672181 25032 ts_manager.cc:194] Registered new tserver with Master: 0bed09c4520749cd8df02207be7afd93 (127.3.137.66:41127)
I20260809 06:24:06.672951 25032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.66:33667
W20260809 06:24:07.003682 25384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:07.003973 25384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:07.004061 25384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:07.011682 25384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:07.011792 25384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:07.011904 25384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.137.67
I20260809 06:24:07.021078 25384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.137.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.137.67
--webserver_port=0
--tserver_master_addrs=127.3.137.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.137.67
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:07.022318 25384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:07.024176 25384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:07.037189 25390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:07.037362 25391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:07.037659 25393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:07.038578 25384 server_base.cc:1034] running on GCE node
I20260809 06:24:07.039283 25384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:07.040210 25384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:07.041462 25384 hybrid_clock.cc:648] HybridClock initialized: now 1786256647041411 us; error 41 us; skew 500 ppm
I20260809 06:24:07.041849 25384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:24:07.044152 25384 webserver.cc:460] Webserver started at http://127.3.137.67:44841/ using document root <none> and password file <none>
I20260809 06:24:07.044823 25384 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:07.044957 25384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:07.045307 25384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:07.048190 25384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data/instance:
uuid: "b17e8cd9a32041c0af6414ab836a5ca5"
format_stamp: "Formatted at 2026-08-09 06:24:07 on dist-test-slave-2w3w"
server_key: "d4d59d2aa7315b47176709944f6f1e3b"
server_key_iv: "d4c410fe087228b4de0026ce8e4ae9f1"
server_key_version: "encryptionkey@0"
I20260809 06:24:07.048987 25384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance:
uuid: "b17e8cd9a32041c0af6414ab836a5ca5"
format_stamp: "Formatted at 2026-08-09 06:24:07 on dist-test-slave-2w3w"
server_key: "d4d59d2aa7315b47176709944f6f1e3b"
server_key_iv: "d4c410fe087228b4de0026ce8e4ae9f1"
server_key_version: "encryptionkey@0"
I20260809 06:24:07.057431 25384 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.009s
I20260809 06:24:07.061980 25399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:07.063277 25384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:24:07.063463 25384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal
uuid: "b17e8cd9a32041c0af6414ab836a5ca5"
format_stamp: "Formatted at 2026-08-09 06:24:07 on dist-test-slave-2w3w"
server_key: "d4d59d2aa7315b47176709944f6f1e3b"
server_key_iv: "d4c410fe087228b4de0026ce8e4ae9f1"
server_key_version: "encryptionkey@0"
I20260809 06:24:07.063702 25384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:07.084463 25384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:07.085263 25384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:07.085577 25384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:07.086627 25384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:24:07.088193 25384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:24:07.088315 25384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:07.088440 25384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:24:07.088533 25384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:07.123718 25384 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.67:37169
I20260809 06:24:07.123816 25524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.67:37169 every 8 connection(s)
I20260809 06:24:07.125106 25384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/data/info.pb
I20260809 06:24:07.130848 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25384
I20260809 06:24:07.131049 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal/instance
I20260809 06:24:07.131513 3621 external_mini_cluster.cc:1428] Setting key feffb7008d1b716d3d4d23be65453411
I20260809 06:24:07.137742 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.127752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.67:60453 (local address 127.3.137.126:42153)
0809 06:24:07.127935 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.127940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.129292 (+ 1352us) server_negotiation.cc:407] Connection header received
0809 06:24:07.131079 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.131087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.131181 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.131296 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:07.132516 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.133526 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.135130 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.135535 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.135676 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.136928 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:07.136932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:07.136936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:07.136959 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:07.136991 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:07.137003 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:07.137100 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.137319 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.137414 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:24:07.138648 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.128250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:42153 (local address 127.3.137.67:60453)
0809 06:24:07.129130 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.129170 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.130789 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.131522 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.131530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.131637 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:07.132341 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.132362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.133687 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.133691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.134904 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.134916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.135759 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.135767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.136002 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.136620 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:24:07.136623 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:07.136658 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:24:07.136733 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:07.137101 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:07.137114 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:07.137124 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.137304 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.137693 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":276,"threads_started":1}
I20260809 06:24:07.140034 25527 heartbeater.cc:347] Connected to a master server at 127.3.137.126:42153
I20260809 06:24:07.140305 25527 heartbeater.cc:464] Registering TS with master...
I20260809 06:24:07.140952 25527 heartbeater.cc:511] Master 127.3.137.126:42153 requested a full tablet report, sending...
I20260809 06:24:07.142335 25032 ts_manager.cc:194] Registered new tserver with Master: b17e8cd9a32041c0af6414ab836a5ca5 (127.3.137.67:37169)
I20260809 06:24:07.143477 25032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.137.67:60453
I20260809 06:24:07.146551 3621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:24:07.156543 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.150541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.3.137.126:42153)
0809 06:24:07.150728 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.150732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.151328 (+ 596us) server_negotiation.cc:407] Connection header received
0809 06:24:07.151451 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.151455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.151521 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.151632 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:07.152155 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.152878 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.154231 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.154667 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.154819 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.155855 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:07.155860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:07.155864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:07.155881 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:07.155910 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:07.155922 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:07.156031 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.156170 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.156259 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:24:07.160105 25032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:24:07.166944 25032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:24:07.197033 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.178702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38636)
0809 06:24:07.187451 (+ 8749us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.187500 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.187769 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.188881 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.188892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.188934 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:07.189560 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.189582 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.192231 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.192235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.194064 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.194079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.196025 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.196032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.196238 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.196296 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.196445 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.196652 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8481,"mutex_wait_us":1,"thread_start_us":8348,"threads_started":1}
I20260809 06:24:07.197012 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.178844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.3.137.65:43419)
0809 06:24:07.186816 (+ 7972us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.186829 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.187608 (+ 779us) server_negotiation.cc:407] Connection header received
0809 06:24:07.188086 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.188095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.188359 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.188529 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:07.190320 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.192025 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.194279 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.195809 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.195978 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.196269 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.196507 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.196668 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7771,"thread_start_us":7668,"threads_started":1}
I20260809 06:24:07.198156 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.179526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.3.137.67:37169)
0809 06:24:07.187994 (+ 8468us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.188002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.188188 (+ 186us) server_negotiation.cc:407] Connection header received
0809 06:24:07.188328 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.188339 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.188565 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.188793 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:07.190353 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.192036 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.194519 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.195967 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.196101 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.196372 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.196922 (+ 550us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.197104 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8278,"thread_start_us":8122,"threads_started":1}
I20260809 06:24:07.198547 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.178912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:37169 (local address 127.0.0.1:60332)
0809 06:24:07.188085 (+ 9173us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.188102 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.188205 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.189167 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.189176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.189214 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:07.190034 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.190048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.192186 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.192194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.194302 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.194317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.196501 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.196508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.196698 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.196734 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.197952 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.198090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9055,"mutex_wait_us":363,"thread_start_us":8951,"threads_started":1}
I20260809 06:24:07.198580 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.188467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41127 (local address 127.0.0.1:60370)
0809 06:24:07.190528 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.190547 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.190656 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.191317 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.191321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.191345 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:07.191695 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.191705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.193627 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.193632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.195830 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.195847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.197733 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.197741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.197915 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.197947 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.198037 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.198153 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1939,"mutex_wait_us":1,"thread_start_us":1245,"threads_started":1}
I20260809 06:24:07.199258 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.188542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.3.137.66:41127)
0809 06:24:07.190271 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.190289 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.190778 (+ 489us) server_negotiation.cc:407] Connection header received
0809 06:24:07.190880 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.190890 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.191140 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.191325 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:07.191849 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.193446 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.196067 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.197505 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.197655 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.197893 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.198706 (+ 813us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.198830 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":1466,"threads_started":1}
I20260809 06:24:07.201663 25165 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8002711a154361bd439fdcb9c467f3 (DEFAULT_TABLE table=test-table [id=aa48d670fc054baca610af8e8e7f3e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:07.203068 25165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8002711a154361bd439fdcb9c467f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:07.202719 25449 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8002711a154361bd439fdcb9c467f3 (DEFAULT_TABLE table=test-table [id=aa48d670fc054baca610af8e8e7f3e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:07.202719 25308 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8002711a154361bd439fdcb9c467f3 (DEFAULT_TABLE table=test-table [id=aa48d670fc054baca610af8e8e7f3e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:24:07.204101 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8002711a154361bd439fdcb9c467f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:07.204099 25449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8002711a154361bd439fdcb9c467f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:07.208590 25234 heartbeater.cc:503] Master 127.3.137.126:42153 was elected leader, sending a full tablet report...
I20260809 06:24:07.215081 25546 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Bootstrap starting.
I20260809 06:24:07.218642 25546 tablet_bootstrap.cc:654] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:07.219169 25548 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Bootstrap starting.
I20260809 06:24:07.220121 25546 log.cc:824] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:07.220845 25547 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Bootstrap starting.
I20260809 06:24:07.223248 25548 tablet_bootstrap.cc:654] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:07.224033 25547 tablet_bootstrap.cc:654] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:07.224448 25548 log.cc:824] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:07.225111 25547 log.cc:824] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:07.225736 25546 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: No bootstrap required, opened a new log
I20260809 06:24:07.226023 25546 ts_tablet_manager.cc:1397] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260809 06:24:07.227618 25548 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: No bootstrap required, opened a new log
I20260809 06:24:07.227902 25548 ts_tablet_manager.cc:1397] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 06:24:07.228353 25547 tablet_bootstrap.cc:492] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: No bootstrap required, opened a new log
I20260809 06:24:07.228662 25547 ts_tablet_manager.cc:1397] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260809 06:24:07.233704 25546 raft_consensus.cc:348] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.234064 25546 raft_consensus.cc:374] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:07.234193 25546 raft_consensus.cc:729] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2115d85d17664e3fb01425ab0380007c, State: Initialized, Role: FOLLOWER
I20260809 06:24:07.234807 25546 consensus_queue.cc:260] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.236517 25546 ts_tablet_manager.cc:1428] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260809 06:24:07.238469 25548 raft_consensus.cc:348] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.238873 25548 raft_consensus.cc:374] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:07.239017 25548 raft_consensus.cc:729] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bed09c4520749cd8df02207be7afd93, State: Initialized, Role: FOLLOWER
I20260809 06:24:07.239293 25547 raft_consensus.cc:348] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.239703 25547 raft_consensus.cc:374] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:07.239799 25547 raft_consensus.cc:729] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17e8cd9a32041c0af6414ab836a5ca5, State: Initialized, Role: FOLLOWER
I20260809 06:24:07.239619 25548 consensus_queue.cc:260] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.240458 25547 consensus_queue.cc:260] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.242580 25527 heartbeater.cc:503] Master 127.3.137.126:42153 was elected leader, sending a full tablet report...
I20260809 06:24:07.242913 25380 heartbeater.cc:503] Master 127.3.137.126:42153 was elected leader, sending a full tablet report...
I20260809 06:24:07.243274 25547 ts_tablet_manager.cc:1428] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260809 06:24:07.243530 25548 ts_tablet_manager.cc:1428] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260809 06:24:07.376940 25528 tablet.cc:2369] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:24:07.407104 25381 tablet.cc:2369] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:24:07.431591 25235 tablet.cc:2369] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:07.583112 25552 raft_consensus.cc:482] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:24:07.583438 25552 raft_consensus.cc:504] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.585203 25552 leader_election.cc:288] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17e8cd9a32041c0af6414ab836a5ca5 (127.3.137.67:37169), 0bed09c4520749cd8df02207be7afd93 (127.3.137.66:41127)
I20260809 06:24:07.592685 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.585837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:55065 (local address 127.3.137.67:37169)
0809 06:24:07.586066 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.586075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.586271 (+ 196us) server_negotiation.cc:407] Connection header received
0809 06:24:07.586559 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.586566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.586658 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.586760 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:07.587574 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.588555 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.590628 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.591663 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.591788 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.591838 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.592278 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.592424 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:24:07.592731 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.585427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.67:37169 (local address 127.3.137.65:55065)
0809 06:24:07.586147 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.586175 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.586392 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.586889 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.586892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.586917 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:07.587411 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.587427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.588696 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.588702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.590418 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.590431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.591858 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.591864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.592093 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.592131 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.592209 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.592309 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":81,"thread_start_us":316,"threads_started":1}
I20260809 06:24:07.592798 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.586544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.137.65:38305 (local address 127.3.137.66:41127)
0809 06:24:07.586763 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.586768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.587145 (+ 377us) server_negotiation.cc:407] Connection header received
0809 06:24:07.587280 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.587286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.587389 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.587499 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:24:07.588122 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.588938 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.590820 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.591843 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.591994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.592065 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.592394 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.592525 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 06:24:07.593060 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.586187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.66:41127 (local address 127.3.137.65:38305)
0809 06:24:07.587037 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.587054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:07.587168 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:07.587532 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:07.587536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:07.587563 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:24:07.587963 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.587976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.589092 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.589098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.590616 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:07.590631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.592042 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.592047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:07.592221 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.592255 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:24:07.592349 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 06:24:07.592464 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":4,"thread_start_us":482,"threads_started":1}
I20260809 06:24:07.593779 25474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8002711a154361bd439fdcb9c467f3" candidate_uuid: "2115d85d17664e3fb01425ab0380007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" is_pre_election: true
I20260809 06:24:07.594157 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8002711a154361bd439fdcb9c467f3" candidate_uuid: "2115d85d17664e3fb01425ab0380007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bed09c4520749cd8df02207be7afd93" is_pre_election: true
I20260809 06:24:07.594393 25474 raft_consensus.cc:2454] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2115d85d17664e3fb01425ab0380007c in term 0.
I20260809 06:24:07.594749 25330 raft_consensus.cc:2454] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2115d85d17664e3fb01425ab0380007c in term 0.
I20260809 06:24:07.595168 25114 leader_election.cc:302] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2115d85d17664e3fb01425ab0380007c, b17e8cd9a32041c0af6414ab836a5ca5; no voters:
I20260809 06:24:07.595633 25552 raft_consensus.cc:2781] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:24:07.595809 25552 raft_consensus.cc:482] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:24:07.595937 25552 raft_consensus.cc:3037] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:07.598219 25552 raft_consensus.cc:504] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.598912 25552 leader_election.cc:288] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [CANDIDATE]: Term 1 election: Requested vote from peers b17e8cd9a32041c0af6414ab836a5ca5 (127.3.137.67:37169), 0bed09c4520749cd8df02207be7afd93 (127.3.137.66:41127)
I20260809 06:24:07.599347 25474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8002711a154361bd439fdcb9c467f3" candidate_uuid: "2115d85d17664e3fb01425ab0380007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17e8cd9a32041c0af6414ab836a5ca5"
I20260809 06:24:07.599426 25330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8002711a154361bd439fdcb9c467f3" candidate_uuid: "2115d85d17664e3fb01425ab0380007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bed09c4520749cd8df02207be7afd93"
I20260809 06:24:07.599556 25474 raft_consensus.cc:3037] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:07.599663 25330 raft_consensus.cc:3037] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:07.601922 25474 raft_consensus.cc:2454] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2115d85d17664e3fb01425ab0380007c in term 1.
I20260809 06:24:07.601925 25330 raft_consensus.cc:2454] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2115d85d17664e3fb01425ab0380007c in term 1.
I20260809 06:24:07.602329 25114 leader_election.cc:302] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bed09c4520749cd8df02207be7afd93, 2115d85d17664e3fb01425ab0380007c; no voters:
I20260809 06:24:07.602648 25552 raft_consensus.cc:2781] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:07.603168 25552 raft_consensus.cc:686] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 1 LEADER]: Becoming Leader. State: Replica: 2115d85d17664e3fb01425ab0380007c, State: Running, Role: LEADER
I20260809 06:24:07.603641 25552 consensus_queue.cc:237] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } }
I20260809 06:24:07.608474 25032 catalog_manager.cc:5495] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2115d85d17664e3fb01425ab0380007c (127.3.137.65). New cstate: current_term: 1 leader_uuid: "2115d85d17664e3fb01425ab0380007c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2115d85d17664e3fb01425ab0380007c" member_type: VOTER last_known_addr { host: "127.3.137.65" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 } health_report { overall_health: UNKNOWN } } }
I20260809 06:24:07.642423 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.636896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.3.137.65:43419)
0809 06:24:07.637103 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.637111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.637195 (+ 84us) server_negotiation.cc:407] Connection header received
0809 06:24:07.637469 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.637473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.637560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.637716 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:07.638143 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.638841 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.640241 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.640652 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.640830 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.641335 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:07.641857 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:07.642017 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.642075 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.642153 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:24:07.650663 25330 raft_consensus.cc:1261] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 1 FOLLOWER]: Refusing update from remote peer 2115d85d17664e3fb01425ab0380007c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:07.650722 25474 raft_consensus.cc:1261] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 1 FOLLOWER]: Refusing update from remote peer 2115d85d17664e3fb01425ab0380007c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:07.651654 25558 consensus_queue.cc:1035] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bed09c4520749cd8df02207be7afd93" member_type: VOTER last_known_addr { host: "127.3.137.66" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:07.652088 25552 consensus_queue.cc:1035] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17e8cd9a32041c0af6414ab836a5ca5" member_type: VOTER last_known_addr { host: "127.3.137.67" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:07.664438 25561 mvcc.cc:205] Tried to move back new op lower bound from 7316507228767019008 to 7316507228588269568. Current Snapshot: MvccSnapshot[applied={T|T < 7316507228767019008}]
I20260809 06:24:07.684809 25030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:24:07.685164 25030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:24:07.689352 25030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3bdafd1975e48ddb4e9f92454a39c46: Sending DeleteTablet for 3 replicas of tablet 4f8002711a154361bd439fdcb9c467f3
I20260809 06:24:07.690704 25449 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8002711a154361bd439fdcb9c467f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:07 UTC) from {username='slave'} at 127.0.0.1:60332
I20260809 06:24:07.690677 25165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8002711a154361bd439fdcb9c467f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:07 UTC) from {username='slave'} at 127.0.0.1:38636
I20260809 06:24:07.690812 25308 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8002711a154361bd439fdcb9c467f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:24:07 UTC) from {username='slave'} at 127.0.0.1:60370
I20260809 06:24:07.692219 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25094
I20260809 06:24:07.704533 25572 tablet_replica.cc:330] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: stopping tablet replica
I20260809 06:24:07.705060 25572 raft_consensus.cc:2229] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:24:07.705708 25572 raft_consensus.cc:2258] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:24:07.708065 25572 ts_tablet_manager.cc:1905] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:07.715516 25572 ts_tablet_manager.cc:1918] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:24:07.715772 25572 log.cc:1196] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-1/wal/wals/4f8002711a154361bd439fdcb9c467f3
I20260809 06:24:07.716413 25572 ts_tablet_manager.cc:1939] T 4f8002711a154361bd439fdcb9c467f3 P 0bed09c4520749cd8df02207be7afd93: Deleting consensus metadata
I20260809 06:24:07.718321 25017 catalog_manager.cc:4841] TS 0bed09c4520749cd8df02207be7afd93 (127.3.137.66:41127): tablet 4f8002711a154361bd439fdcb9c467f3 (table test-table [id=aa48d670fc054baca610af8e8e7f3e69]) successfully deleted
I20260809 06:24:07.720373 25571 tablet_replica.cc:330] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: stopping tablet replica
I20260809 06:24:07.720965 25571 raft_consensus.cc:2229] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:24:07.721807 25571 raft_consensus.cc:2258] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:24:07.725222 25570 tablet_replica.cc:330] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: stopping tablet replica
I20260809 06:24:07.725608 25570 raft_consensus.cc:2229] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:24:07.726060 25570 raft_consensus.cc:2258] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:24:07.727099 25571 ts_tablet_manager.cc:1905] T 4f8002711a154361bd439fdcb9c467f3 P 2115d85d17664e3fb01425ab0380007c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:07.727986 25570 ts_tablet_manager.cc:1905] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:24:07.731204 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.693974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.3.137.65:43419)
0809 06:24:07.694203 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.694212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.717159 (+ 22947us) server_negotiation.cc:407] Connection header received
0809 06:24:07.717744 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.717753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.717941 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.718039 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:07.718943 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.720235 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.722459 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.723206 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.723315 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.729406 (+ 6091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:07.729421 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:07.729437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:07.730215 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:07.730281 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:07.730308 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:07.730388 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.730637 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.730754 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:24:07.732824 25229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:07.735785 25570 ts_tablet_manager.cc:1918] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:24:07.735984 25570 log.cc:1196] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Deleting WAL directory at /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786256324325922-3621-0/minicluster-data/ts-2/wal/wals/4f8002711a154361bd439fdcb9c467f3
I20260809 06:24:07.736581 25570 ts_tablet_manager.cc:1939] T 4f8002711a154361bd439fdcb9c467f3 P b17e8cd9a32041c0af6414ab836a5ca5: Deleting consensus metadata
I20260809 06:24:07.738075 25017 catalog_manager.cc:4841] TS b17e8cd9a32041c0af6414ab836a5ca5 (127.3.137.67:37169): tablet 4f8002711a154361bd439fdcb9c467f3 (table test-table [id=aa48d670fc054baca610af8e8e7f3e69]) successfully deleted
I20260809 06:24:07.832680 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25094
I20260809 06:24:07.853216 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25239
W20260809 06:24:07.853209 25016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.137.65:43419 (error 108)
I20260809 06:24:07.855279 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.854417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38682)
0809 06:24:07.854892 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.855045 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":199,"threads_started":1}
W20260809 06:24:07.855557 25016 catalog_manager.cc:4584] TS 2115d85d17664e3fb01425ab0380007c (127.3.137.65:43419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f8002711a154361bd439fdcb9c467f3: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
I20260809 06:24:07.860291 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.854042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.3.137.66:41127)
0809 06:24:07.854254 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.854260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.854355 (+ 95us) server_negotiation.cc:407] Connection header received
0809 06:24:07.854510 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.854519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.854655 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.854788 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:07.855343 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.856182 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.857415 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.857905 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.857999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.859080 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:07.859087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:07.859101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:07.859665 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:07.859713 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:07.859732 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:07.859838 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.859946 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.860026 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:24:07.860898 25374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:07.909138 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.908665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38692)
0809 06:24:07.908889 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.908971 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 06:24:07.910156 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.909791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38706)
0809 06:24:07.909935 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.910025 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 06:24:07.949355 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25239
I20260809 06:24:07.968200 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25384
I20260809 06:24:07.970815 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.970316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38714)
0809 06:24:07.970528 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.970631 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:24:07.971865 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.971455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38718)
0809 06:24:07.971640 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:07.971707 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:24:07.976114 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:07.969095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.3.137.67:37169)
0809 06:24:07.969287 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:07.969295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:07.969318 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:24:07.969551 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:07.969557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:07.969659 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:07.969787 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:07.970242 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.971329 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.972780 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:07.973221 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:07.973368 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:07.974568 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:07.974578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:07.974593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:07.975306 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:07.975357 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:07.975377 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:07.975518 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:07.975655 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 06:24:07.975766 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:24:07.976795 25519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:08.052613 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:08.052045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38720)
0809 06:24:08.052299 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:08.052411 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 06:24:08.053812 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:08.053328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.65:43419 (local address 127.0.0.1:38726)
0809 06:24:08.053552 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:08.053638 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.65:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 06:24:08.065996 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25384
I20260809 06:24:08.085335 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 24997
I20260809 06:24:08.086526 25064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:08.194752 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 24997
2026-08-09T06:24:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3145 ms)
[----------] 6 tests from EncryptionPolicyTest (22365 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:08 dist-test-slave-2w3w krb5kdc[25584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:08 dist-test-slave-2w3w krb5kdc[25584](info): set up 2 sockets
Aug 09 06:24:08 dist-test-slave-2w3w krb5kdc[25584](info): commencing operation
krb5kdc: starting...
W20260809 06:24:09.364683 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:09 dist-test-slave-2w3w krb5kdc[25584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:09.672852 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:10.009018 25599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:10.009330 25599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:10.009497 25599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:10.016862 25599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:10.016980 25599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:10.017076 25599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:10.017190 25599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:10.017251 25599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:10.027304 25599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:10.028582 25599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:10.029855 25599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:10.041664 25604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:10.041630 25605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:10.041767 25607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:10.042335 25599 server_base.cc:1034] running on GCE node
I20260809 06:24:10.043298 25599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:10.044729 25599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:10.045926 25599 hybrid_clock.cc:648] HybridClock initialized: now 1786256650045876 us; error 64 us; skew 500 ppm
I20260809 06:24:10.046257 25599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:10 dist-test-slave-2w3w krb5kdc[25584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:10.049381 25599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:10.050823 25599 webserver.cc:460] Webserver started at http://127.0.0.1:40831/ using document root <none> and password file <none>
I20260809 06:24:10.051515 25599 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:10.051649 25599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:10.051983 25599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:10.054853 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3"
format_stamp: "Formatted at 2026-08-09 06:24:10 on dist-test-slave-2w3w"
server_key: "cdf59f470962ca8e382241710012adc1"
server_key_iv: "50be78d7623726e7db650b92cb07a521"
server_key_version: "encryptionkey@0"
I20260809 06:24:10.055714 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3"
format_stamp: "Formatted at 2026-08-09 06:24:10 on dist-test-slave-2w3w"
server_key: "cdf59f470962ca8e382241710012adc1"
server_key_iv: "50be78d7623726e7db650b92cb07a521"
server_key_version: "encryptionkey@0"
I20260809 06:24:10.060708 25599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:24:10.063988 25615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:10.065189 25599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:10.065372 25599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3"
format_stamp: "Formatted at 2026-08-09 06:24:10 on dist-test-slave-2w3w"
server_key: "cdf59f470962ca8e382241710012adc1"
server_key_iv: "50be78d7623726e7db650b92cb07a521"
server_key_version: "encryptionkey@0"
I20260809 06:24:10.065589 25599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:10.101228 25599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:10.105180 25599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:10.105492 25599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:10.124264 25599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34319
I20260809 06:24:10.124312 25669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34319 every 8 connection(s)
I20260809 06:24:10.125670 25599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:10.129364 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:10.131568 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25599
I20260809 06:24:10.131793 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:10.132509 3621 external_mini_cluster.cc:1428] Setting key e7dfb56d2348e0a412086b5b2a3887eb
I20260809 06:24:10.141351 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3: Bootstrap starting.
I20260809 06:24:10.145071 25670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:10.146421 25670 log.cc:824] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:24:10 dist-test-slave-2w3w krb5kdc[25584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:24:10.149696 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3: No bootstrap required, opened a new log
I20260809 06:24:10.156473 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:10.134463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47401 (local address 127.0.0.1:34319)
0809 06:24:10.135284 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:10.135295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:10.135336 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:24:10.136931 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:10.136962 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:10.137331 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:10.137645 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:10.139530 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:10.140799 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:10.142645 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:10.143448 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:10.143578 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:10.148899 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:10.148906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:10.148920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:10.148961 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:10.152199 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:10.153114 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:10.153124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:10.153130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:10.153294 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:10.153818 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:10.153821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:10.153823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:10.154296 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:10.154545 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:10.155156 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 06:24:10.155486 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":430,"threads_started":1}
I20260809 06:24:10.159132 25670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } }
I20260809 06:24:10.159536 25670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:10.159657 25670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd0b9f9ebe64696a2aac7d9769e75a3, State: Initialized, Role: FOLLOWER
I20260809 06:24:10.160326 25670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } }
I20260809 06:24:10.160619 25670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:10.160665 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:10.160771 25670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:10.160960 25670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:10.163229 25670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } }
I20260809 06:24:10.163748 25670 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd0b9f9ebe64696a2aac7d9769e75a3; no voters:
I20260809 06:24:10.164464 25670 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:10.164543 25679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:10.167168 25670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:10.167284 25679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd0b9f9ebe64696a2aac7d9769e75a3, State: Running, Role: LEADER
I20260809 06:24:10.167958 25679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } }
I20260809 06:24:10.171083 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } } }
I20260809 06:24:10.171461 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:10.172009 25681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd0b9f9ebe64696a2aac7d9769e75a3. Latest consensus state: current_term: 1 leader_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd0b9f9ebe64696a2aac7d9769e75a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34319 } } }
I20260809 06:24:10.172323 25681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:10.182914 25688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:10.186560 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:10.166276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51965 (local address 127.0.0.1:34319)
0809 06:24:10.166662 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:10.166667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:10.166685 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:24:10.166744 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:10.166748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:10.166802 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:10.166919 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:10.168622 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:10.170047 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:10.171701 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:10.172701 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:10.172863 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:10.177180 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:10.177190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:10.177198 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:10.177244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:10.182138 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:10.183284 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:10.183291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:10.183298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:10.183475 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:10.184109 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:10.184117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:10.184120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:10.184641 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:10.185016 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:10.185991 (+ 975us) server_negotiation.cc:299] Negotiation successful
0809 06:24:10.186186 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260809 06:24:10.188674 25688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:10.197575 25688 catalog_manager.cc:1347] Generated new cluster ID: 34f3b353169a4634adb7350795c55cb3
I20260809 06:24:10.197693 25688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:10.216764 25688 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:10.218722 25688 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:10.228150 25688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd0b9f9ebe64696a2aac7d9769e75a3: Generated new TSK 0
I20260809 06:24:10.228940 25688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:10.242525 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25599
I20260809 06:24:10.244712 25665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:10.335911 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25599
W20260809 06:24:10.356106 25582 connection.cc:535] client connection to 127.0.0.1:34319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34319): Connection reset by peer (error 104)
2026-08-09T06:24:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:10 dist-test-slave-2w3w krb5kdc[25705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:10 dist-test-slave-2w3w krb5kdc[25705](info): set up 2 sockets
Aug 09 06:24:10 dist-test-slave-2w3w krb5kdc[25705](info): commencing operation
krb5kdc: starting...
W20260809 06:24:11.517349 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:11 dist-test-slave-2w3w krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:11.822822 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:12.156230 25719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:12.156502 25719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:12.156663 25719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:12.163832 25719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:12.163934 25719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:12.164049 25719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:12.164166 25719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:12.164232 25719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:12.173797 25719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:12.175104 25719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:12.176384 25719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:12.188097 25725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:12.188215 25726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:12.188176 25728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:12.189049 25719 server_base.cc:1034] running on GCE node
I20260809 06:24:12.189806 25719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:12.191257 25719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:12.192507 25719 hybrid_clock.cc:648] HybridClock initialized: now 1786256652192414 us; error 79 us; skew 500 ppm
I20260809 06:24:12.192889 25719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:12 dist-test-slave-2w3w krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:12.196177 25719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:12.197674 25719 webserver.cc:460] Webserver started at http://127.0.0.1:34855/ using document root <none> and password file <none>
I20260809 06:24:12.198439 25719 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:12.198596 25719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:12.198907 25719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:12.202016 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "700306afea514458bcb9bf49943e6532"
format_stamp: "Formatted at 2026-08-09 06:24:12 on dist-test-slave-2w3w"
server_key: "6b7b3f7a512d5e7c8122511e6aa30273"
server_key_iv: "b91138b0a2c358d72f2bf60bb7f700de"
server_key_version: "encryptionkey@0"
I20260809 06:24:12.202823 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "700306afea514458bcb9bf49943e6532"
format_stamp: "Formatted at 2026-08-09 06:24:12 on dist-test-slave-2w3w"
server_key: "6b7b3f7a512d5e7c8122511e6aa30273"
server_key_iv: "b91138b0a2c358d72f2bf60bb7f700de"
server_key_version: "encryptionkey@0"
I20260809 06:24:12.208170 25719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:24:12.211634 25735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:12.212944 25719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:24:12.213140 25719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "700306afea514458bcb9bf49943e6532"
format_stamp: "Formatted at 2026-08-09 06:24:12 on dist-test-slave-2w3w"
server_key: "6b7b3f7a512d5e7c8122511e6aa30273"
server_key_iv: "b91138b0a2c358d72f2bf60bb7f700de"
server_key_version: "encryptionkey@0"
I20260809 06:24:12.213347 25719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:12.246927 25719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:12.251459 25719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:12.251756 25719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:12.270685 25719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43409
I20260809 06:24:12.270737 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43409 every 8 connection(s)
I20260809 06:24:12.272075 25719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:12.273288 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25719
I20260809 06:24:12.273489 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:12.274041 3621 external_mini_cluster.cc:1428] Setting key 415115507b077456ab087b3440892859
I20260809 06:24:12.276595 25795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:12.287444 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:12.275876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59111 (local address 127.0.0.1:43409)
0809 06:24:12.276710 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:12.276725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:12.276778 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:24:12.278373 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:12.278394 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:12.278739 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:12.279163 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:12.279721 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:12.281050 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:12.283037 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:12.283686 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:12.283809 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:12.285130 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:12.285138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:12.285153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:12.285702 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:12.285780 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:12.285806 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:12.285928 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:12.286271 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 06:24:12.286483 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":364,"threads_started":1}
I20260809 06:24:12.288548 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532: Bootstrap starting.
I20260809 06:24:12.291090 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:12.291823 25795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:12.293017 25795 log.cc:824] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:12.296312 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532: No bootstrap required, opened a new log
I20260809 06:24:12.303369 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:12.296379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47327 (local address 127.0.0.1:43409)
0809 06:24:12.296569 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:12.296577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:12.296996 (+ 419us) server_negotiation.cc:407] Connection header received
0809 06:24:12.297172 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:12.297175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:12.297231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:12.297306 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:12.297932 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:12.298904 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:12.300862 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:12.301275 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:12.301390 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:12.302536 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:12.302540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:12.302543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:12.302558 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:12.302610 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:12.302625 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:12.302716 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:12.302941 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 06:24:12.303042 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:24:12.305795 25795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260809 06:24:12.306167 25795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:12.306277 25795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700306afea514458bcb9bf49943e6532, State: Initialized, Role: FOLLOWER
I20260809 06:24:12.306824 25795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260809 06:24:12.307140 25795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:12.307293 25795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:12.307487 25795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:12.309536 25795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260809 06:24:12.310030 25795 leader_election.cc:302] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700306afea514458bcb9bf49943e6532; no voters:
I20260809 06:24:12.310684 25795 leader_election.cc:288] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:12.310822 25807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:12.311538 25807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [term 1 LEADER]: Becoming Leader. State: Replica: 700306afea514458bcb9bf49943e6532, State: Running, Role: LEADER
I20260809 06:24:12.312158 25807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260809 06:24:12.313057 25795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:12.315315 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700306afea514458bcb9bf49943e6532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } } }
I20260809 06:24:12.315752 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:12.314949 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700306afea514458bcb9bf49943e6532. Latest consensus state: current_term: 1 leader_uuid: "700306afea514458bcb9bf49943e6532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700306afea514458bcb9bf49943e6532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } } }
I20260809 06:24:12.316267 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:12.317755 25814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:12.322937 25814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:12.330799 25814 catalog_manager.cc:1347] Generated new cluster ID: ee4ddef84c874389a99a28d11afcf63a
I20260809 06:24:12.330910 25814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:12.343966 25814 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:12.345728 25814 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:12.352922 25814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700306afea514458bcb9bf49943e6532: Generated new TSK 0
I20260809 06:24:12.353865 25814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:12.361660 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25719
W20260809 06:24:12.363202 25790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59111
W20260809 06:24:12.363981 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:12.364163 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25719
W20260809 06:24:12.383795 25703 connection.cc:535] client connection to 127.0.0.1:43409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43409): Connection reset by peer (error 104)
2026-08-09T06:24:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:12 dist-test-slave-2w3w krb5kdc[25830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:12 dist-test-slave-2w3w krb5kdc[25830](info): set up 2 sockets
Aug 09 06:24:12 dist-test-slave-2w3w krb5kdc[25830](info): commencing operation
krb5kdc: starting...
W20260809 06:24:13.521126 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:13 dist-test-slave-2w3w krb5kdc[25830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:13.831317 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:14.167572 25845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:14.167846 25845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:14.167940 25845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:14.175001 25845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:14.175096 25845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:14.175141 25845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:14.175201 25845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:14.175237 25845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:14.184661 25845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:14.185788 25845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:14.187060 25845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:14.198954 25851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:14.198921 25850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:14.199132 25853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:14.200073 25845 server_base.cc:1034] running on GCE node
I20260809 06:24:14.200874 25845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:14.202394 25845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:14.203750 25845 hybrid_clock.cc:648] HybridClock initialized: now 1786256654203672 us; error 71 us; skew 500 ppm
I20260809 06:24:14.204071 25845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:14 dist-test-slave-2w3w krb5kdc[25830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:14.212824 25845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:14.214461 25845 webserver.cc:460] Webserver started at http://127.0.0.1:40813/ using document root <none> and password file <none>
I20260809 06:24:14.215293 25845 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:14.215438 25845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:14.215764 25845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:14.218825 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "c1964e8e8f244bbbb20c294b22c12317"
format_stamp: "Formatted at 2026-08-09 06:24:14 on dist-test-slave-2w3w"
server_key: "5eb76e3bd850dad41d706731b4b49f0c"
server_key_iv: "b04d285c9e7a3da9d98a0c0130c88134"
server_key_version: "encryptionkey@0"
I20260809 06:24:14.219697 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "c1964e8e8f244bbbb20c294b22c12317"
format_stamp: "Formatted at 2026-08-09 06:24:14 on dist-test-slave-2w3w"
server_key: "5eb76e3bd850dad41d706731b4b49f0c"
server_key_iv: "b04d285c9e7a3da9d98a0c0130c88134"
server_key_version: "encryptionkey@0"
I20260809 06:24:14.224846 25845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:24:14.228237 25860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:14.229455 25845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:24:14.229647 25845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "c1964e8e8f244bbbb20c294b22c12317"
format_stamp: "Formatted at 2026-08-09 06:24:14 on dist-test-slave-2w3w"
server_key: "5eb76e3bd850dad41d706731b4b49f0c"
server_key_iv: "b04d285c9e7a3da9d98a0c0130c88134"
server_key_version: "encryptionkey@0"
I20260809 06:24:14.229835 25845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:14.235684 25845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:14.239639 25845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:14.239946 25845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:14.259112 25845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39165
I20260809 06:24:14.259150 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39165 every 8 connection(s)
I20260809 06:24:14.260484 25845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:14.261569 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25845
I20260809 06:24:14.261793 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:14.262254 3621 external_mini_cluster.cc:1428] Setting key 749d4411f27af0fe375a4d1b9e9eb526
I20260809 06:24:14.264884 25917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:14.271265 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:14.264135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45311 (local address 127.0.0.1:39165)
0809 06:24:14.264990 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:14.265004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:14.265054 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:24:14.267474 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:14.267500 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:14.267938 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:14.268378 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:14.269355 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:14.269363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:14.269380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:14.269410 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:14.269499 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:14.269532 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:14.269736 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:14.269865 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 06:24:14.270065 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":339,"threads_started":1}
I20260809 06:24:14.274780 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:14.277478 25917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317: Bootstrap starting.
I20260809 06:24:14.280835 25917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:14.281584 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:14.279177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36147 (local address 127.0.0.1:39165)
0809 06:24:14.279349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:14.279355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:14.279705 (+ 350us) server_negotiation.cc:407] Connection header received
0809 06:24:14.279819 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:14.279824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:14.279880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:14.279985 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:14.280882 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:14.280887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:14.280893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:14.280914 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:14.280963 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:14.280983 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:14.281098 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:14.281229 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 06:24:14.281303 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:24:14.282114 25917 log.cc:824] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:14.284852 25917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317: No bootstrap required, opened a new log
I20260809 06:24:14.291962 25917 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } }
I20260809 06:24:14.292340 25917 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:14.292450 25917 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1964e8e8f244bbbb20c294b22c12317, State: Initialized, Role: FOLLOWER
I20260809 06:24:14.293004 25917 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } }
I20260809 06:24:14.293251 25917 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:14.293401 25917 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:14.293576 25917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:14.295809 25917 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } }
I20260809 06:24:14.296296 25917 leader_election.cc:302] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1964e8e8f244bbbb20c294b22c12317; no voters:
I20260809 06:24:14.296877 25917 leader_election.cc:288] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:14.297017 25928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:14.297742 25928 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [term 1 LEADER]: Becoming Leader. State: Replica: c1964e8e8f244bbbb20c294b22c12317, State: Running, Role: LEADER
I20260809 06:24:14.298337 25928 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } }
I20260809 06:24:14.299082 25917 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:14.301468 25930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1964e8e8f244bbbb20c294b22c12317. Latest consensus state: current_term: 1 leader_uuid: "c1964e8e8f244bbbb20c294b22c12317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } } }
I20260809 06:24:14.301939 25930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:14.303325 25929 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1964e8e8f244bbbb20c294b22c12317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1964e8e8f244bbbb20c294b22c12317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39165 } } }
I20260809 06:24:14.303642 25929 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:14.307631 25935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:14.311906 25935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:14.320765 25935 catalog_manager.cc:1347] Generated new cluster ID: 12711a9b79af47b2972abb1b0f4ff7b8
I20260809 06:24:14.320890 25935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:14.333060 25935 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:14.333473 25935 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:14.340463 25935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1964e8e8f244bbbb20c294b22c12317: Generated new TSK 0
I20260809 06:24:14.341396 25935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:14.353256 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25845
W20260809 06:24:14.354698 25911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45311
W20260809 06:24:14.355537 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25845: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:14.355654 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25845
2026-08-09T06:24:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:14 dist-test-slave-2w3w krb5kdc[25951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:14 dist-test-slave-2w3w krb5kdc[25951](info): set up 2 sockets
Aug 09 06:24:14 dist-test-slave-2w3w krb5kdc[25951](info): commencing operation
krb5kdc: starting...
W20260809 06:24:15.505055 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:15 dist-test-slave-2w3w krb5kdc[25951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:15.815140 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:16.148907 25965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:16.149180 25965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:16.149274 25965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:16.156414 25965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:16.156522 25965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:16.156569 25965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:16.156638 25965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:16.156682 25965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:16.166159 25965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:16.167346 25965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:16.168578 25965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:16.180470 25971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:16.180522 25973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:16.180702 25970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:16.181349 25965 server_base.cc:1034] running on GCE node
I20260809 06:24:16.182166 25965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:16.183669 25965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:16.184873 25965 hybrid_clock.cc:648] HybridClock initialized: now 1786256656184818 us; error 56 us; skew 500 ppm
I20260809 06:24:16.185214 25965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:16 dist-test-slave-2w3w krb5kdc[25951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:16.191793 25965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:16.193410 25965 webserver.cc:460] Webserver started at http://127.0.0.1:37837/ using document root <none> and password file <none>
I20260809 06:24:16.194104 25965 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:16.194242 25965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:16.194566 25965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:16.197750 25965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "288e1cfd2ccf4a0181f311619e6617db"
format_stamp: "Formatted at 2026-08-09 06:24:16 on dist-test-slave-2w3w"
server_key: "ad4f0b1fae6303757458d91f22866e8b"
server_key_iv: "32995a59bf30b70e4d8ce457e4d21706"
server_key_version: "encryptionkey@0"
I20260809 06:24:16.198516 25965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "288e1cfd2ccf4a0181f311619e6617db"
format_stamp: "Formatted at 2026-08-09 06:24:16 on dist-test-slave-2w3w"
server_key: "ad4f0b1fae6303757458d91f22866e8b"
server_key_iv: "32995a59bf30b70e4d8ce457e4d21706"
server_key_version: "encryptionkey@0"
I20260809 06:24:16.203794 25965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:24:16.207332 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:16.208645 25965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:24:16.208864 25965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "288e1cfd2ccf4a0181f311619e6617db"
format_stamp: "Formatted at 2026-08-09 06:24:16 on dist-test-slave-2w3w"
server_key: "ad4f0b1fae6303757458d91f22866e8b"
server_key_iv: "32995a59bf30b70e4d8ce457e4d21706"
server_key_version: "encryptionkey@0"
I20260809 06:24:16.209066 25965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:16.234766 25965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:16.238746 25965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:16.239079 25965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:16.263298 25965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36491
I20260809 06:24:16.263319 26040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36491 every 8 connection(s)
I20260809 06:24:16.265273 25965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:16.270285 26041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:16.274893 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 25965
I20260809 06:24:16.275107 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:16.275772 3621 external_mini_cluster.cc:1428] Setting key 876521358449295f5e72f33508ac44a1
I20260809 06:24:16.286901 26041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db: Bootstrap starting.
I20260809 06:24:16.291882 26041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db: Neither blocks nor log segments found. Creating new log.
Aug 09 06:24:16 dist-test-slave-2w3w krb5kdc[25951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:24:16.293339 26041 log.cc:824] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:16.297152 26041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db: No bootstrap required, opened a new log
I20260809 06:24:16.301648 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:16.277789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57421 (local address 127.0.0.1:36491)
0809 06:24:16.278757 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:16.278781 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:16.278851 (+ 70us) server_negotiation.cc:407] Connection header received
0809 06:24:16.281398 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:16.281425 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:16.281961 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:16.282383 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:16.284062 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:16.285355 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:16.286725 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:16.287786 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:16.288003 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:16.293599 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:16.293607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:16.293621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:16.293660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:16.297249 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:16.297965 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:16.297973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:16.297981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:16.298162 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:16.298595 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:16.298604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:16.298608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:16.299227 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:16.299573 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:16.300025 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 06:24:16.300611 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":424,"threads_started":1}
I20260809 06:24:16.305411 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:16.305199 26041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } }
I20260809 06:24:16.305585 26041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:16.305681 26041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288e1cfd2ccf4a0181f311619e6617db, State: Initialized, Role: FOLLOWER
I20260809 06:24:16.306318 26041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } }
I20260809 06:24:16.306566 26041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:16.306730 26041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:16.306895 26041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:16.309080 26041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } }
I20260809 06:24:16.309574 26041 leader_election.cc:302] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 288e1cfd2ccf4a0181f311619e6617db; no voters:
I20260809 06:24:16.310286 26041 leader_election.cc:288] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:16.310403 26051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:16.311151 26051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [term 1 LEADER]: Becoming Leader. State: Replica: 288e1cfd2ccf4a0181f311619e6617db, State: Running, Role: LEADER
I20260809 06:24:16.311906 26051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } }
I20260809 06:24:16.316206 26051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 288e1cfd2ccf4a0181f311619e6617db. Latest consensus state: current_term: 1 leader_uuid: "288e1cfd2ccf4a0181f311619e6617db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } } }
I20260809 06:24:16.316728 26051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:16.320842 26054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "288e1cfd2ccf4a0181f311619e6617db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e1cfd2ccf4a0181f311619e6617db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36491 } } }
I20260809 06:24:16.321266 26054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:16.324810 26041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:16.326871 26056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:16.331173 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:16.310249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42233 (local address 127.0.0.1:36491)
0809 06:24:16.310480 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:16.310493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:16.311039 (+ 546us) server_negotiation.cc:407] Connection header received
0809 06:24:16.311329 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:16.311334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:16.311399 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:16.311465 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:16.313402 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:16.315862 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:16.317730 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:16.318748 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:16.318952 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:16.324217 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:16.324226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:16.324234 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:16.324284 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:16.328154 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:16.328961 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:16.328969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:16.328976 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:16.329156 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:16.329570 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:16.329576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:16.329579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:16.329866 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:16.330157 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:16.330518 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 06:24:16.330742 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 06:24:16.339681 26056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:16.351054 26056 catalog_manager.cc:1347] Generated new cluster ID: f421fea1619c4c0ab9b48492590b112a
I20260809 06:24:16.351183 26056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:16.370304 26056 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:16.372181 26056 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:16.380893 26056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 288e1cfd2ccf4a0181f311619e6617db: Generated new TSK 0
I20260809 06:24:16.381651 26056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:16.394845 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 25965
I20260809 06:24:16.396018 26036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:16.490135 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 25965
2026-08-09T06:24:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:16 dist-test-slave-2w3w krb5kdc[26075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:16 dist-test-slave-2w3w krb5kdc[26075](info): set up 2 sockets
Aug 09 06:24:16 dist-test-slave-2w3w krb5kdc[26075](info): commencing operation
krb5kdc: starting...
W20260809 06:24:17.653105 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:17 dist-test-slave-2w3w krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:17.966666 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:18.303005 26089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:18.303313 26089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:18.303483 26089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:18.310500 26089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:18.310636 26089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:18.310750 26089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:18.310860 26089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:18.310928 26089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:18.320737 26089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:18.322000 26089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:18.323323 26089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:18.335347 26094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:18.335297 26097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:18.335372 26095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:18.336386 26089 server_base.cc:1034] running on GCE node
I20260809 06:24:18.337200 26089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:18.338543 26089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:18.339838 26089 hybrid_clock.cc:648] HybridClock initialized: now 1786256658339765 us; error 58 us; skew 500 ppm
I20260809 06:24:18.340205 26089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:18 dist-test-slave-2w3w krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:18.343400 26089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:18.344797 26089 webserver.cc:460] Webserver started at http://127.0.0.1:34205/ using document root <none> and password file <none>
I20260809 06:24:18.345481 26089 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:18.345629 26089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:18.345961 26089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:18.348867 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "254ba317851d48618deda843b3f6f091"
format_stamp: "Formatted at 2026-08-09 06:24:18 on dist-test-slave-2w3w"
server_key: "ed694dcad4082b6646ff2b4ba0e7d234"
server_key_iv: "3cf00eb2ea951b474c4e550c132b227e"
server_key_version: "encryptionkey@0"
I20260809 06:24:18.349620 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "254ba317851d48618deda843b3f6f091"
format_stamp: "Formatted at 2026-08-09 06:24:18 on dist-test-slave-2w3w"
server_key: "ed694dcad4082b6646ff2b4ba0e7d234"
server_key_iv: "3cf00eb2ea951b474c4e550c132b227e"
server_key_version: "encryptionkey@0"
I20260809 06:24:18.354548 26089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:24:18.357978 26106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:18.359309 26089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:24:18.359494 26089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "254ba317851d48618deda843b3f6f091"
format_stamp: "Formatted at 2026-08-09 06:24:18 on dist-test-slave-2w3w"
server_key: "ed694dcad4082b6646ff2b4ba0e7d234"
server_key_iv: "3cf00eb2ea951b474c4e550c132b227e"
server_key_version: "encryptionkey@0"
I20260809 06:24:18.359686 26089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:18.374760 26089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:18.379104 26089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:18.379388 26089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:18.397346 26089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45017
I20260809 06:24:18.397421 26158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45017 every 8 connection(s)
I20260809 06:24:18.398615 26089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:18.402216 26159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:18.406054 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26089
I20260809 06:24:18.406236 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:18.406677 3621 external_mini_cluster.cc:1428] Setting key c74367e0fe22014c6cd501618acdf81e
I20260809 06:24:18.413067 26159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091: Bootstrap starting.
I20260809 06:24:18.416570 26159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:18.417802 26159 log.cc:824] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:18.420706 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:18.408396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42437 (local address 127.0.0.1:45017)
0809 06:24:18.409137 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:18.409147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:18.409190 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:24:18.411527 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:18.411548 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:18.411982 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:18.412386 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:18.413010 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:18.414348 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:18.416256 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:18.416971 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:18.417100 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:18.418312 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:18.418343 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:18.418363 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:18.419164 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:18.419266 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:18.419291 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:18.419374 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:18.419591 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 06:24:18.419858 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":366,"threads_started":1}
I20260809 06:24:18.420799 26159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091: No bootstrap required, opened a new log
I20260809 06:24:18.424629 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:18.431155 26159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260809 06:24:18.431656 26159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:18.431798 26159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254ba317851d48618deda843b3f6f091, State: Initialized, Role: FOLLOWER
I20260809 06:24:18.432421 26159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260809 06:24:18.432729 26159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:18.432891 26159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:18.433087 26159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:18.435951 26159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260809 06:24:18.436560 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:18.429773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55155 (local address 127.0.0.1:45017)
0809 06:24:18.429953 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:18.429959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:18.430445 (+ 486us) server_negotiation.cc:407] Connection header received
0809 06:24:18.430580 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:18.430585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:18.430654 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:18.430761 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:18.431268 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:18.432356 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:18.433871 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:18.434600 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:18.434752 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:18.435802 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:18.435810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:18.435819 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:18.435842 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:18.435889 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:18.435910 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:18.436018 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:18.436157 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 06:24:18.436242 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:24:18.436542 26159 leader_election.cc:302] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254ba317851d48618deda843b3f6f091; no voters:
I20260809 06:24:18.437318 26159 leader_election.cc:288] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:18.437429 26173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:18.439224 26173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [term 1 LEADER]: Becoming Leader. State: Replica: 254ba317851d48618deda843b3f6f091, State: Running, Role: LEADER
I20260809 06:24:18.439961 26159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:18.439827 26173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260809 06:24:18.442574 26174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254ba317851d48618deda843b3f6f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } } }
I20260809 06:24:18.443017 26174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:18.443531 26175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254ba317851d48618deda843b3f6f091. Latest consensus state: current_term: 1 leader_uuid: "254ba317851d48618deda843b3f6f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254ba317851d48618deda843b3f6f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } } }
I20260809 06:24:18.443892 26175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:18.443958 26182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:18.449402 26182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:18.457885 26182 catalog_manager.cc:1347] Generated new cluster ID: 3cb004c6e0674b1b888c3d50b17606d1
I20260809 06:24:18.458010 26182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:18.477124 26182 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:18.479108 26182 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:18.494052 26182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254ba317851d48618deda843b3f6f091: Generated new TSK 0
I20260809 06:24:18.495160 26182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:18.502971 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26089
W20260809 06:24:18.504041 26154 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42437
W20260809 06:24:18.504719 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26089: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:18.504876 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26089
2026-08-09T06:24:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:18 dist-test-slave-2w3w krb5kdc[26197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:18 dist-test-slave-2w3w krb5kdc[26197](info): set up 2 sockets
Aug 09 06:24:18 dist-test-slave-2w3w krb5kdc[26197](info): commencing operation
krb5kdc: starting...
W20260809 06:24:19.664600 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.005s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:19 dist-test-slave-2w3w krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:19.956096 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:20.287829 26210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:20.288131 26210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:20.288308 26210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:20.295380 26210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:20.295506 26210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:20.295609 26210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:20.295725 26210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:20.295792 26210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:20.305413 26210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:20.306655 26210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:20.307988 26210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:20.319844 26215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:20.319903 26218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:20.319936 26216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:20.320981 26210 server_base.cc:1034] running on GCE node
I20260809 06:24:20.321724 26210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:20.323069 26210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:20.324294 26210 hybrid_clock.cc:648] HybridClock initialized: now 1786256660324266 us; error 39 us; skew 500 ppm
I20260809 06:24:20.324669 26210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:20 dist-test-slave-2w3w krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:20.330929 26210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:20.332368 26210 webserver.cc:460] Webserver started at http://127.0.0.1:45731/ using document root <none> and password file <none>
I20260809 06:24:20.333039 26210 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:20.333187 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:20.333485 26210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:20.336382 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "ae707b1028fb4e90a326dbea4d6a977f"
format_stamp: "Formatted at 2026-08-09 06:24:20 on dist-test-slave-2w3w"
server_key: "b8e6df12d56aff0cbb6e2325430b457c"
server_key_iv: "b8c3cd3bd98cfcf62e02fdbe12164949"
server_key_version: "encryptionkey@0"
I20260809 06:24:20.337147 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "ae707b1028fb4e90a326dbea4d6a977f"
format_stamp: "Formatted at 2026-08-09 06:24:20 on dist-test-slave-2w3w"
server_key: "b8e6df12d56aff0cbb6e2325430b457c"
server_key_iv: "b8c3cd3bd98cfcf62e02fdbe12164949"
server_key_version: "encryptionkey@0"
I20260809 06:24:20.342020 26210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 06:24:20.345314 26225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:20.346560 26210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:24:20.346742 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "ae707b1028fb4e90a326dbea4d6a977f"
format_stamp: "Formatted at 2026-08-09 06:24:20 on dist-test-slave-2w3w"
server_key: "b8e6df12d56aff0cbb6e2325430b457c"
server_key_iv: "b8c3cd3bd98cfcf62e02fdbe12164949"
server_key_version: "encryptionkey@0"
I20260809 06:24:20.346936 26210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:20.352326 26210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:20.356062 26210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:20.356370 26210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:20.375337 26210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42207
I20260809 06:24:20.375418 26280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42207 every 8 connection(s)
I20260809 06:24:20.376703 26210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:20.380276 26281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:20.385105 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26210
I20260809 06:24:20.385303 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:20.385828 3621 external_mini_cluster.cc:1428] Setting key 92ccf538ff40d5269144090f69216f56
I20260809 06:24:20.391398 26281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f: Bootstrap starting.
I20260809 06:24:20.393314 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:20.387587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34057 (local address 127.0.0.1:42207)
0809 06:24:20.388395 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:20.388406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:20.388448 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:24:20.390165 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:20.390184 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:20.390494 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:20.390798 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:20.391877 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:20.391883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:20.391895 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:20.391915 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:20.391970 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:20.391989 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:20.392154 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:20.392274 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 06:24:20.392445 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":403,"threads_started":1}
I20260809 06:24:20.394726 26281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:20.395932 26281 log.cc:824] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:20.397142 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:20.398582 26281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f: No bootstrap required, opened a new log
I20260809 06:24:20.404106 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:20.401604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42789 (local address 127.0.0.1:42207)
0809 06:24:20.401810 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:20.401817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:20.402158 (+ 341us) server_negotiation.cc:407] Connection header received
0809 06:24:20.402308 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:20.402311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:20.402352 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:20.402422 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:20.403439 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:20.403443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:20.403446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:20.403470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:20.403508 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:20.403523 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:20.403617 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:20.403740 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 06:24:20.403782 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 06:24:20.407327 26281 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 06:24:20.407667 26281 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:20.407783 26281 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae707b1028fb4e90a326dbea4d6a977f, State: Initialized, Role: FOLLOWER
I20260809 06:24:20.408335 26281 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 06:24:20.408571 26281 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:20.408717 26281 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:20.408922 26281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:20.411537 26281 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 06:24:20.412034 26281 leader_election.cc:302] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae707b1028fb4e90a326dbea4d6a977f; no voters:
I20260809 06:24:20.412587 26281 leader_election.cc:288] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:20.412724 26292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:20.413389 26292 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [term 1 LEADER]: Becoming Leader. State: Replica: ae707b1028fb4e90a326dbea4d6a977f, State: Running, Role: LEADER
I20260809 06:24:20.413990 26292 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } }
I20260809 06:24:20.414568 26281 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:20.417117 26294 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae707b1028fb4e90a326dbea4d6a977f. Latest consensus state: current_term: 1 leader_uuid: "ae707b1028fb4e90a326dbea4d6a977f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } } }
I20260809 06:24:20.417053 26293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae707b1028fb4e90a326dbea4d6a977f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae707b1028fb4e90a326dbea4d6a977f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42207 } } }
I20260809 06:24:20.417524 26294 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:20.417562 26293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:20.419463 26298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:20.424508 26298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:20.435463 26298 catalog_manager.cc:1347] Generated new cluster ID: b9c0f12719f74a2e9f439029805447ac
I20260809 06:24:20.435575 26298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:20.465227 26298 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:20.465727 26298 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:20.475808 26298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae707b1028fb4e90a326dbea4d6a977f: Generated new TSK 0
I20260809 06:24:20.476751 26298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:20.491711 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26210
W20260809 06:24:20.492839 26276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34057
W20260809 06:24:20.493665 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:20.493827 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26210
2026-08-09T06:24:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:20 dist-test-slave-2w3w krb5kdc[26314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:20 dist-test-slave-2w3w krb5kdc[26314](info): set up 2 sockets
Aug 09 06:24:20 dist-test-slave-2w3w krb5kdc[26314](info): commencing operation
krb5kdc: starting...
W20260809 06:24:21.652204 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:21 dist-test-slave-2w3w krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:21.958942 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:22.301774 26328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:22.302155 26328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:22.302368 26328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:22.310654 26328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:22.310765 26328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:22.310858 26328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:22.310963 26328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:22.311065 26328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:22.321187 26328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:22.322456 26328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:22.323839 26328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:22.336562 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:22.336745 26333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:22.336710 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:22.337414 26328 server_base.cc:1034] running on GCE node
I20260809 06:24:22.338385 26328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:22.340179 26328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:22.341441 26328 hybrid_clock.cc:648] HybridClock initialized: now 1786256662341359 us; error 85 us; skew 500 ppm
I20260809 06:24:22.341899 26328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:22 dist-test-slave-2w3w krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:22.345944 26328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:22.347501 26328 webserver.cc:460] Webserver started at http://127.0.0.1:39667/ using document root <none> and password file <none>
I20260809 06:24:22.348168 26328 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:22.348321 26328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:22.348625 26328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:22.351614 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "bb97a2b81cdd4b018f0869516f97441e"
format_stamp: "Formatted at 2026-08-09 06:24:22 on dist-test-slave-2w3w"
server_key: "0780c98a0b8bdeca31e722eeadc66b28"
server_key_iv: "dfd9ef1ffd701aa22859e5cddaaa44d3"
server_key_version: "encryptionkey@0"
I20260809 06:24:22.352396 26328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "bb97a2b81cdd4b018f0869516f97441e"
format_stamp: "Formatted at 2026-08-09 06:24:22 on dist-test-slave-2w3w"
server_key: "0780c98a0b8bdeca31e722eeadc66b28"
server_key_iv: "dfd9ef1ffd701aa22859e5cddaaa44d3"
server_key_version: "encryptionkey@0"
I20260809 06:24:22.357270 26328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 06:24:22.360524 26348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:22.361733 26328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 06:24:22.361908 26328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "bb97a2b81cdd4b018f0869516f97441e"
format_stamp: "Formatted at 2026-08-09 06:24:22 on dist-test-slave-2w3w"
server_key: "0780c98a0b8bdeca31e722eeadc66b28"
server_key_iv: "dfd9ef1ffd701aa22859e5cddaaa44d3"
server_key_version: "encryptionkey@0"
I20260809 06:24:22.362098 26328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:22.382479 26328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:22.386214 26328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:22.386503 26328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:22.405093 26328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43533
I20260809 06:24:22.405134 26403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43533 every 8 connection(s)
I20260809 06:24:22.406418 26328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:22.409971 26404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:22.410701 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26328
I20260809 06:24:22.410907 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:22.411401 3621 external_mini_cluster.cc:1428] Setting key 2daae3a021a1f4e01bcd08c487ec4102
I20260809 06:24:22.424238 26404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e: Bootstrap starting.
Aug 09 06:24:22 dist-test-slave-2w3w krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:24:22.427215 26404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:22.428435 26404 log.cc:824] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:22.431864 26404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e: No bootstrap required, opened a new log
I20260809 06:24:22.436756 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:22.413370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57319 (local address 127.0.0.1:43533)
0809 06:24:22.414133 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:22.414149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:22.414194 (+ 45us) server_negotiation.cc:407] Connection header received
0809 06:24:22.416653 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:22.416680 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:22.417187 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:22.417622 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:22.418981 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:22.421071 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:22.422445 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:22.423480 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:22.423661 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:22.427981 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:22.427992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:22.428012 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:22.428061 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:22.431971 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:22.432815 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:22.432824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:22.432831 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:22.433065 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:22.433527 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:22.433537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:22.433541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:22.434245 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:22.434596 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:22.435077 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 06:24:22.435408 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
I20260809 06:24:22.440722 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:22.441802 26404 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260809 06:24:22.442162 26404 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:22.442266 26404 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb97a2b81cdd4b018f0869516f97441e, State: Initialized, Role: FOLLOWER
I20260809 06:24:22.442806 26404 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260809 06:24:22.443104 26404 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:22.443274 26404 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:22.443439 26404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:22.445909 26404 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260809 06:24:22.446501 26404 leader_election.cc:302] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb97a2b81cdd4b018f0869516f97441e; no voters:
I20260809 06:24:22.447198 26404 leader_election.cc:288] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:22.447256 26416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:22.448559 26416 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [term 1 LEADER]: Becoming Leader. State: Replica: bb97a2b81cdd4b018f0869516f97441e, State: Running, Role: LEADER
I20260809 06:24:22.449307 26404 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:22.449216 26416 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260809 06:24:22.452299 26417 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb97a2b81cdd4b018f0869516f97441e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260809 06:24:22.453200 26417 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:22.452874 26418 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb97a2b81cdd4b018f0869516f97441e. Latest consensus state: current_term: 1 leader_uuid: "bb97a2b81cdd4b018f0869516f97441e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb97a2b81cdd4b018f0869516f97441e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260809 06:24:22.453716 26418 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:22.455870 26422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:22.461932 26422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:22.465420 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:22.445279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:42073 (local address 127.0.0.1:43533)
0809 06:24:22.445481 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:22.445490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:22.445995 (+ 505us) server_negotiation.cc:407] Connection header received
0809 06:24:22.446206 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:22.446210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:22.446267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:22.446349 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:22.448349 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:22.449571 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:22.451638 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:22.452561 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:22.452713 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:22.456875 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:22.456883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:22.456887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:22.456922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:22.460996 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:22.462046 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:22.462052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:22.462056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:22.462194 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:22.462807 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:22.462812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:22.462816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:22.463316 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:22.464654 (+ 1338us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:22.464830 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 06:24:22.465021 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:24:22.473941 26422 catalog_manager.cc:1347] Generated new cluster ID: 19e5899193ff490bb17fc35195364951
I20260809 06:24:22.474095 26422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:22.486340 26422 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:22.487756 26422 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:22.502230 26422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb97a2b81cdd4b018f0869516f97441e: Generated new TSK 0
I20260809 06:24:22.503317 26422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:22.518014 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26328
I20260809 06:24:22.519722 26398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:22.612819 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26328
W20260809 06:24:22.632933 26312 connection.cc:535] client connection to 127.0.0.1:43533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43533): Connection reset by peer (error 104)
2026-08-09T06:24:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:22 dist-test-slave-2w3w krb5kdc[26440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:22 dist-test-slave-2w3w krb5kdc[26440](info): set up 2 sockets
Aug 09 06:24:22 dist-test-slave-2w3w krb5kdc[26440](info): commencing operation
krb5kdc: starting...
W20260809 06:24:23.769590 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:23 dist-test-slave-2w3w krb5kdc[26440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:24.082432 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:24.412956 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:24.413270 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:24.413432 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:24.420143 26453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:24.420303 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:24.420411 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:24.420539 26453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:24.420629 26453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:24.430436 26453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:24.431680 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:24.433043 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:24.444744 26459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:24.444743 26460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:24.444752 26462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:24.445712 26453 server_base.cc:1034] running on GCE node
I20260809 06:24:24.446493 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:24.447829 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:24.449059 26453 hybrid_clock.cc:648] HybridClock initialized: now 1786256664449023 us; error 46 us; skew 500 ppm
I20260809 06:24:24.449388 26453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:24 dist-test-slave-2w3w krb5kdc[26440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:24.452484 26453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:24.453796 26453 webserver.cc:460] Webserver started at http://127.0.0.1:33335/ using document root <none> and password file <none>
I20260809 06:24:24.454377 26453 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:24.454497 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:24.454748 26453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:24.457548 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "6dbbaa75b129477d8a5e0383dff18d66"
format_stamp: "Formatted at 2026-08-09 06:24:24 on dist-test-slave-2w3w"
server_key: "3c3843767ac0aada595f035ef4784496"
server_key_iv: "b94146527586067b6394dbf0abffd3b6"
server_key_version: "encryptionkey@0"
I20260809 06:24:24.458232 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "6dbbaa75b129477d8a5e0383dff18d66"
format_stamp: "Formatted at 2026-08-09 06:24:24 on dist-test-slave-2w3w"
server_key: "3c3843767ac0aada595f035ef4784496"
server_key_iv: "b94146527586067b6394dbf0abffd3b6"
server_key_version: "encryptionkey@0"
I20260809 06:24:24.463011 26453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260809 06:24:24.466207 26469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:24.467562 26453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:24.467690 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "6dbbaa75b129477d8a5e0383dff18d66"
format_stamp: "Formatted at 2026-08-09 06:24:24 on dist-test-slave-2w3w"
server_key: "3c3843767ac0aada595f035ef4784496"
server_key_iv: "b94146527586067b6394dbf0abffd3b6"
server_key_version: "encryptionkey@0"
I20260809 06:24:24.467840 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:24.484925 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:24.489107 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:24.489341 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:24.508116 26453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42423
I20260809 06:24:24.508165 26525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42423 every 8 connection(s)
I20260809 06:24:24.509472 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:24.513132 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:24.519819 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26453
I20260809 06:24:24.519999 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:24.520462 3621 external_mini_cluster.cc:1428] Setting key 1612695c50ea80f073752974de526ebc
I20260809 06:24:24.523898 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66: Bootstrap starting.
I20260809 06:24:24.527400 26526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:24.528637 26526 log.cc:824] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:24.531883 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66: No bootstrap required, opened a new log
I20260809 06:24:24.534200 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:24.522376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:44291 (local address 127.0.0.1:42423)
0809 06:24:24.523125 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:24.523136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:24.523173 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:24:24.524909 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:24.524930 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:24.525436 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:24.525769 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:24.526390 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:24.527798 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:24.529677 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:24.530446 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:24.530592 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:24.531885 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:24.531892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:24.531904 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:24.532533 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:24.532612 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:24.532639 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:24.532754 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:24.533079 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 06:24:24.533256 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
I20260809 06:24:24.537940 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:24.539835 26526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260809 06:24:24.540169 26526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:24.540318 26526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbbaa75b129477d8a5e0383dff18d66, State: Initialized, Role: FOLLOWER
I20260809 06:24:24.541005 26526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260809 06:24:24.541298 26526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:24.541457 26526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:24.541651 26526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:24.544721 26526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260809 06:24:24.545228 26526 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbbaa75b129477d8a5e0383dff18d66; no voters:
I20260809 06:24:24.545960 26526 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:24.546057 26537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:24.547194 26537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbbaa75b129477d8a5e0383dff18d66, State: Running, Role: LEADER
I20260809 06:24:24.547837 26537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260809 06:24:24.548355 26526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:24.549477 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:24.542370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:56003 (local address 127.0.0.1:42423)
0809 06:24:24.542602 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:24.542611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:24.543119 (+ 508us) server_negotiation.cc:407] Connection header received
0809 06:24:24.543330 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:24.543333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:24.543395 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:24.543494 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:24.544110 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:24.544834 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:24.546468 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:24.547204 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:24.547345 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:24.548643 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:24.548649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:24.548653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:24.548679 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:24.548734 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:24.548749 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:24.548888 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:24.549085 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 06:24:24.549140 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:24:24.553134 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dbbaa75b129477d8a5e0383dff18d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260809 06:24:24.553107 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbbaa75b129477d8a5e0383dff18d66. Latest consensus state: current_term: 1 leader_uuid: "6dbbaa75b129477d8a5e0383dff18d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbbaa75b129477d8a5e0383dff18d66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260809 06:24:24.553622 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:24.553694 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:24.554541 26546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:24.569010 26546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:24.584471 26546 catalog_manager.cc:1347] Generated new cluster ID: e811772ace014bbba72500bd1e5e36d9
I20260809 06:24:24.584619 26546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:24.594739 26546 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:24.596069 26546 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:24.608611 26546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbbaa75b129477d8a5e0383dff18d66: Generated new TSK 0
I20260809 06:24:24.609443 26546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:24.616106 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26453
W20260809 06:24:24.617381 26521 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:44291
W20260809 06:24:24.618098 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:24.618247 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26453
W20260809 06:24:24.638104 26437 connection.cc:535] client connection to 127.0.0.1:42423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42423): Connection reset by peer (error 104)
2026-08-09T06:24:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:24 dist-test-slave-2w3w krb5kdc[26559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:24 dist-test-slave-2w3w krb5kdc[26559](info): set up 2 sockets
Aug 09 06:24:24 dist-test-slave-2w3w krb5kdc[26559](info): commencing operation
krb5kdc: starting...
W20260809 06:24:25.788463 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:25 dist-test-slave-2w3w krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:26.094261 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:26.439532 26573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:26.439806 26573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:26.440001 26573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:26.447007 26573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:26.447130 26573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:26.447211 26573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:26.447322 26573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:26.447407 26573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:26.457010 26573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:26.458266 26573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:26.459589 26573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:26.471549 26578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:26.471499 26582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:26.471446 26579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:26.472371 26573 server_base.cc:1034] running on GCE node
I20260809 06:24:26.473114 26573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:26.474445 26573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:26.475696 26573 hybrid_clock.cc:648] HybridClock initialized: now 1786256666475653 us; error 49 us; skew 500 ppm
I20260809 06:24:26.476087 26573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:26 dist-test-slave-2w3w krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:26.479228 26573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:26.480691 26573 webserver.cc:460] Webserver started at http://127.0.0.1:37599/ using document root <none> and password file <none>
I20260809 06:24:26.481362 26573 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:26.481513 26573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:26.481827 26573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:26.484774 26573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "0ee22685671a4855a30dc86dfaa1a200"
format_stamp: "Formatted at 2026-08-09 06:24:26 on dist-test-slave-2w3w"
server_key: "a7277a2cbc3d0fd937f5781e3bf07861"
server_key_iv: "dc5f81a9dbc947e64a9d2d9c0892bdb3"
server_key_version: "encryptionkey@0"
I20260809 06:24:26.485554 26573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "0ee22685671a4855a30dc86dfaa1a200"
format_stamp: "Formatted at 2026-08-09 06:24:26 on dist-test-slave-2w3w"
server_key: "a7277a2cbc3d0fd937f5781e3bf07861"
server_key_iv: "dc5f81a9dbc947e64a9d2d9c0892bdb3"
server_key_version: "encryptionkey@0"
I20260809 06:24:26.490604 26573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:24:26.493896 26589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:26.495246 26573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:26.495420 26573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "0ee22685671a4855a30dc86dfaa1a200"
format_stamp: "Formatted at 2026-08-09 06:24:26 on dist-test-slave-2w3w"
server_key: "a7277a2cbc3d0fd937f5781e3bf07861"
server_key_iv: "dc5f81a9dbc947e64a9d2d9c0892bdb3"
server_key_version: "encryptionkey@0"
I20260809 06:24:26.495620 26573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:26.501315 26573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:26.505013 26573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:26.505303 26573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:26.524133 26573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43255
I20260809 06:24:26.524209 26649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43255 every 8 connection(s)
I20260809 06:24:26.525440 26573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:26.529032 26651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:26.533478 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26573
I20260809 06:24:26.533663 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:26.534173 3621 external_mini_cluster.cc:1428] Setting key 8d0d5006961725f31ddf523411da524b
I20260809 06:24:26.540454 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200: Bootstrap starting.
I20260809 06:24:26.542209 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:26.535942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:46023 (local address 127.0.0.1:43255)
0809 06:24:26.536688 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:26.536699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:26.536738 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:24:26.538824 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:26.538845 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:26.539219 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:26.539543 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:26.540656 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:26.540663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:26.540674 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:26.540688 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:26.540745 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:26.540771 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:26.540929 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:26.541066 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 06:24:26.541256 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":338,"threads_started":1}
I20260809 06:24:26.543735 26651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:26.544904 26651 log.cc:824] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:26.545885 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:26.548072 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200: No bootstrap required, opened a new log
I20260809 06:24:26.553987 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:26.551275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:37939 (local address 127.0.0.1:43255)
0809 06:24:26.551430 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:26.551434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:26.551834 (+ 400us) server_negotiation.cc:407] Connection header received
0809 06:24:26.552001 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:26.552005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:26.552051 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:26.552137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:26.553298 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:26.553305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:26.553315 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:26.553335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:26.553387 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:26.553408 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:26.553548 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:26.553601 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 06:24:26.553667 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 06:24:26.557447 26651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260809 06:24:26.557816 26651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:26.557950 26651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee22685671a4855a30dc86dfaa1a200, State: Initialized, Role: FOLLOWER
I20260809 06:24:26.558503 26651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260809 06:24:26.558753 26651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:26.558913 26651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:26.559108 26651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:26.561259 26651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260809 06:24:26.561770 26651 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ee22685671a4855a30dc86dfaa1a200; no voters:
I20260809 06:24:26.562335 26651 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:26.562462 26661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:26.563290 26661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee22685671a4855a30dc86dfaa1a200, State: Running, Role: LEADER
I20260809 06:24:26.563817 26661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260809 06:24:26.564462 26651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:26.566649 26662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ee22685671a4855a30dc86dfaa1a200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } } }
I20260809 06:24:26.567034 26662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:26.568993 26663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee22685671a4855a30dc86dfaa1a200. Latest consensus state: current_term: 1 leader_uuid: "0ee22685671a4855a30dc86dfaa1a200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee22685671a4855a30dc86dfaa1a200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } } }
I20260809 06:24:26.569267 26663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:26.570518 26667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:26.575626 26667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:26.583753 26667 catalog_manager.cc:1347] Generated new cluster ID: df46961de8714e02b921c20505e5078c
I20260809 06:24:26.583877 26667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:26.604378 26667 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:26.604815 26667 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:26.612329 26667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ee22685671a4855a30dc86dfaa1a200: Generated new TSK 0
I20260809 06:24:26.613042 26667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:26.623509 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26573
W20260809 06:24:26.624555 26645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:46023
W20260809 06:24:26.625304 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26573: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:26.625429 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26573
2026-08-09T06:24:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2007 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:26 dist-test-slave-2w3w krb5kdc[26682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:26 dist-test-slave-2w3w krb5kdc[26682](info): set up 2 sockets
Aug 09 06:24:26 dist-test-slave-2w3w krb5kdc[26682](info): commencing operation
krb5kdc: starting...
W20260809 06:24:27.780505 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:27 dist-test-slave-2w3w krb5kdc[26682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:28.074419 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:28.401801 26695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:28.402127 26695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:28.402298 26695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:28.409468 26695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:28.409617 26695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:28.409722 26695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:28.409839 26695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:28.409919 26695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:28.419520 26695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:28.420687 26695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:28.421936 26695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:28.433920 26700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:28.434310 26703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:28.434343 26701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:28.435333 26695 server_base.cc:1034] running on GCE node
I20260809 06:24:28.436146 26695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:28.437454 26695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:28.438656 26695 hybrid_clock.cc:648] HybridClock initialized: now 1786256668438620 us; error 28 us; skew 500 ppm
I20260809 06:24:28.439064 26695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:28 dist-test-slave-2w3w krb5kdc[26682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:28.442178 26695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:28.443604 26695 webserver.cc:460] Webserver started at http://127.0.0.1:35545/ using document root <none> and password file <none>
I20260809 06:24:28.444249 26695 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:28.444389 26695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:28.444684 26695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:28.447569 26695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "ed7062ab4c1b4004a3b89934db17f898"
format_stamp: "Formatted at 2026-08-09 06:24:28 on dist-test-slave-2w3w"
server_key: "4f889f3fd48a419a30e75c6b9ec42c55"
server_key_iv: "b2c6a1288b110688fff6d3b44fea0071"
server_key_version: "encryptionkey@0"
I20260809 06:24:28.448299 26695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "ed7062ab4c1b4004a3b89934db17f898"
format_stamp: "Formatted at 2026-08-09 06:24:28 on dist-test-slave-2w3w"
server_key: "4f889f3fd48a419a30e75c6b9ec42c55"
server_key_iv: "b2c6a1288b110688fff6d3b44fea0071"
server_key_version: "encryptionkey@0"
I20260809 06:24:28.453078 26695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 06:24:28.456434 26710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:28.457661 26695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:24:28.457836 26695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "ed7062ab4c1b4004a3b89934db17f898"
format_stamp: "Formatted at 2026-08-09 06:24:28 on dist-test-slave-2w3w"
server_key: "4f889f3fd48a419a30e75c6b9ec42c55"
server_key_iv: "b2c6a1288b110688fff6d3b44fea0071"
server_key_version: "encryptionkey@0"
I20260809 06:24:28.458024 26695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:28.472069 26695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:28.475859 26695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:28.476130 26695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:28.495194 26695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39537
I20260809 06:24:28.495217 26771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39537 every 8 connection(s)
I20260809 06:24:28.496502 26695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:28.499630 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26695
I20260809 06:24:28.499822 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:28.500380 3621 external_mini_cluster.cc:1428] Setting key 65a2b515fea06bb01acd7641b4ee067f
I20260809 06:24:28.500444 26772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:28.512259 26772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898: Bootstrap starting.
Aug 09 06:24:28 dist-test-slave-2w3w krb5kdc[26682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:24:28.515874 26772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:28.517318 26772 log.cc:824] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:28.520615 26772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898: No bootstrap required, opened a new log
I20260809 06:24:28.523672 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:28.502099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:53745 (local address 127.0.0.1:39537)
0809 06:24:28.502814 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:28.502824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:28.502861 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:24:28.504947 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:28.504967 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:28.505323 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:28.505707 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:28.507235 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:28.508454 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:28.509661 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:28.510324 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:28.510548 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:28.515271 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:28.515278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:28.515291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:28.515340 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:28.519062 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:28.519859 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:28.519867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:28.519875 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:28.520100 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:28.520526 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:28.520531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:28.520534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:28.521255 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:28.521598 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:28.521947 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 06:24:28.522278 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":310,"threads_started":1}
I20260809 06:24:28.527629 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:28.529855 26772 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260809 06:24:28.530231 26772 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:28.530427 26772 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7062ab4c1b4004a3b89934db17f898, State: Initialized, Role: FOLLOWER
I20260809 06:24:28.531270 26772 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260809 06:24:28.531522 26772 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:28.531675 26772 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:28.531857 26772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:28.535259 26772 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260809 06:24:28.535778 26772 leader_election.cc:302] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7062ab4c1b4004a3b89934db17f898; no voters:
I20260809 06:24:28.536450 26772 leader_election.cc:288] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:28.536566 26783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:28.538439 26772 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:28.543818 26783 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [term 1 LEADER]: Becoming Leader. State: Replica: ed7062ab4c1b4004a3b89934db17f898, State: Running, Role: LEADER
I20260809 06:24:28.544602 26783 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260809 06:24:28.548348 26785 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7062ab4c1b4004a3b89934db17f898. Latest consensus state: current_term: 1 leader_uuid: "ed7062ab4c1b4004a3b89934db17f898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260809 06:24:28.548794 26785 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:28.548082 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7062ab4c1b4004a3b89934db17f898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7062ab4c1b4004a3b89934db17f898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260809 06:24:28.549737 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:28.552376 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:28.531992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:45999 (local address 127.0.0.1:39537)
0809 06:24:28.532165 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:28.532171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:28.532728 (+ 557us) server_negotiation.cc:407] Connection header received
0809 06:24:28.532913 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:28.532917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:28.532990 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:28.533076 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:28.534944 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:28.535816 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:28.537780 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:28.538738 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:28.538918 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:28.542789 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:28.542798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:28.542810 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:28.542861 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:28.547231 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:28.548380 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:28.548384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:28.548387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:28.548558 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:28.549224 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:28.549232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:28.549237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:28.549752 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:28.551334 (+ 1582us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:28.551574 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 06:24:28.551822 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:24:28.552639 26798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:28.556900 26798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:28.564640 26798 catalog_manager.cc:1347] Generated new cluster ID: 4505990525ab4b58a2b380542337a545
I20260809 06:24:28.564743 26798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:28.590768 26798 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:28.592109 26798 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:28.599524 26798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7062ab4c1b4004a3b89934db17f898: Generated new TSK 0
I20260809 06:24:28.600286 26798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:28.608897 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26695
I20260809 06:24:28.610070 26767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:28.700488 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26695
W20260809 06:24:28.720039 26680 connection.cc:535] client connection to 127.0.0.1:39537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39537): Connection reset by peer (error 104)
2026-08-09T06:24:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:28 dist-test-slave-2w3w krb5kdc[26808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:28 dist-test-slave-2w3w krb5kdc[26808](info): set up 2 sockets
Aug 09 06:24:28 dist-test-slave-2w3w krb5kdc[26808](info): commencing operation
krb5kdc: starting...
W20260809 06:24:29.857205 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:30 dist-test-slave-2w3w krb5kdc[26808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:30.166152 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:30.497738 26822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:30.498090 26822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:30.498256 26822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:30.505584 26822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:30.505746 26822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:30.505867 26822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:30.505978 26822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:30.506068 26822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:30.516400 26822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:30.517652 26822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:30.518940 26822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:30.530627 26828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:30.530726 26827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:30.530701 26830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:30.531358 26822 server_base.cc:1034] running on GCE node
I20260809 06:24:30.532023 26822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:30.533290 26822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:30.534490 26822 hybrid_clock.cc:648] HybridClock initialized: now 1786256670534444 us; error 45 us; skew 500 ppm
I20260809 06:24:30.534881 26822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:30 dist-test-slave-2w3w krb5kdc[26808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:30.538280 26822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:30.539852 26822 webserver.cc:460] Webserver started at http://127.0.0.1:42301/ using document root <none> and password file <none>
I20260809 06:24:30.540498 26822 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:30.540632 26822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:30.540954 26822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:30.543913 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "958b0f7be142448ea465ba3984331eb6"
format_stamp: "Formatted at 2026-08-09 06:24:30 on dist-test-slave-2w3w"
server_key: "2f5c87eb0957c30213cd66173ec9cab7"
server_key_iv: "ead07ddfc6ff57ce05c4b0e70bbbe4ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:30.544677 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "958b0f7be142448ea465ba3984331eb6"
format_stamp: "Formatted at 2026-08-09 06:24:30 on dist-test-slave-2w3w"
server_key: "2f5c87eb0957c30213cd66173ec9cab7"
server_key_iv: "ead07ddfc6ff57ce05c4b0e70bbbe4ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:30.549544 26822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 06:24:30.552824 26838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:30.554057 26822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:24:30.554227 26822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "958b0f7be142448ea465ba3984331eb6"
format_stamp: "Formatted at 2026-08-09 06:24:30 on dist-test-slave-2w3w"
server_key: "2f5c87eb0957c30213cd66173ec9cab7"
server_key_iv: "ead07ddfc6ff57ce05c4b0e70bbbe4ab"
server_key_version: "encryptionkey@0"
I20260809 06:24:30.554384 26822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:30.568377 26822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:30.572223 26822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:30.572496 26822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:30.591216 26822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37011
I20260809 06:24:30.591254 26895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37011 every 8 connection(s)
I20260809 06:24:30.592581 26822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:30.594941 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26822
I20260809 06:24:30.595202 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:30.595665 3621 external_mini_cluster.cc:1428] Setting key 0576adc1237de92839e74c3d14e3e09d
I20260809 06:24:30.596668 26896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:30.610340 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:30.597469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:37551 (local address 127.0.0.1:37011)
0809 06:24:30.598278 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:30.598293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:30.598342 (+ 49us) server_negotiation.cc:407] Connection header received
0809 06:24:30.600184 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:30.600210 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:30.600728 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:30.601154 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:30.601680 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:30.603798 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:30.605214 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:30.606261 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:30.606436 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:30.607674 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:30.607686 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:30.607704 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:30.608453 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:30.608545 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:30.608594 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:30.608790 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:30.608948 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 06:24:30.609130 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":303,"threads_started":1}
I20260809 06:24:30.611541 26896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: Bootstrap starting.
I20260809 06:24:30.614061 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:30.614805 26896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:30.616185 26896 log.cc:824] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:30.618942 26896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: No bootstrap required, opened a new log
I20260809 06:24:30.626616 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:30.619621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:47481 (local address 127.0.0.1:37011)
0809 06:24:30.619806 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:30.619812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:30.620304 (+ 492us) server_negotiation.cc:407] Connection header received
0809 06:24:30.620445 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:30.620450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:30.620529 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:30.620665 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:30.621165 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:30.622427 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:30.623878 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:30.624652 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:30.624797 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:30.625865 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:30.625872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:30.625876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:30.625896 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:30.625943 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:30.625963 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:30.626089 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:30.626258 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 06:24:30.626306 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:24:30.628980 26896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260809 06:24:30.629335 26896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:30.629439 26896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958b0f7be142448ea465ba3984331eb6, State: Initialized, Role: FOLLOWER
I20260809 06:24:30.630045 26896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260809 06:24:30.630272 26896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:30.630421 26896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:30.630625 26896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:30.632840 26896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260809 06:24:30.633338 26896 leader_election.cc:302] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 958b0f7be142448ea465ba3984331eb6; no voters:
I20260809 06:24:30.633940 26896 leader_election.cc:288] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:30.634032 26907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:30.634802 26907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 958b0f7be142448ea465ba3984331eb6, State: Running, Role: LEADER
I20260809 06:24:30.635396 26907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260809 06:24:30.636050 26896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:30.638374 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "958b0f7be142448ea465ba3984331eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } } }
I20260809 06:24:30.638814 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:30.639429 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 958b0f7be142448ea465ba3984331eb6. Latest consensus state: current_term: 1 leader_uuid: "958b0f7be142448ea465ba3984331eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958b0f7be142448ea465ba3984331eb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } } }
I20260809 06:24:30.639765 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6 [sys.catalog]: This master's current role is: LEADER
W20260809 06:24:30.650638 26926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:24:30.650804 26926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:24:30.650943 26916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:30.652524 26916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:30.661628 26916 catalog_manager.cc:1347] Generated new cluster ID: f90db56cd1fe41bfa2954ab5f425bde1
I20260809 06:24:30.661728 26916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:30.677639 26916 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:30.679448 26916 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:30.700174 26916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 958b0f7be142448ea465ba3984331eb6: Generated new TSK 0
I20260809 06:24:30.701047 26916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:30.708750 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26822
W20260809 06:24:30.710086 26891 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:37551
W20260809 06:24:30.710866 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26822: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:30.711063 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26822
W20260809 06:24:30.731516 26806 connection.cc:535] client connection to 127.0.0.1:37011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37011): Connection reset by peer (error 104)
2026-08-09T06:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:30 dist-test-slave-2w3w krb5kdc[26931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:30 dist-test-slave-2w3w krb5kdc[26931](info): set up 2 sockets
Aug 09 06:24:30 dist-test-slave-2w3w krb5kdc[26931](info): commencing operation
krb5kdc: starting...
W20260809 06:24:31.866150 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:32 dist-test-slave-2w3w krb5kdc[26931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:24:32.174265 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:32.507036 26948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:32.507339 26948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:32.507436 26948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:32.514590 26948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:32.514694 26948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:32.514782 26948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:32.514844 26948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:32.514889 26948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:32.524562 26948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:32.525888 26948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:32.527218 26948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:32.539259 26953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:32.539217 26954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:32.539320 26956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:32.540337 26948 server_base.cc:1034] running on GCE node
I20260809 06:24:32.541126 26948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:32.542572 26948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:32.543773 26948 hybrid_clock.cc:648] HybridClock initialized: now 1786256672543718 us; error 51 us; skew 500 ppm
I20260809 06:24:32.544124 26948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:32 dist-test-slave-2w3w krb5kdc[26931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:32.550465 26948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:24:32.551926 26948 webserver.cc:460] Webserver started at http://127.0.0.1:33465/ using document root <none> and password file <none>
I20260809 06:24:32.552569 26948 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:32.552716 26948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:32.553010 26948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:32.555984 26948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "a6e3d5d709534a27bc15b2f371adbfb8"
format_stamp: "Formatted at 2026-08-09 06:24:32 on dist-test-slave-2w3w"
server_key: "e78c80344f2cebe107cae5faf18dcaf6"
server_key_iv: "edeb86ebe3473eda0561b2d74ecf2500"
server_key_version: "encryptionkey@0"
I20260809 06:24:32.556746 26948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "a6e3d5d709534a27bc15b2f371adbfb8"
format_stamp: "Formatted at 2026-08-09 06:24:32 on dist-test-slave-2w3w"
server_key: "e78c80344f2cebe107cae5faf18dcaf6"
server_key_iv: "edeb86ebe3473eda0561b2d74ecf2500"
server_key_version: "encryptionkey@0"
I20260809 06:24:32.561770 26948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:24:32.565124 26963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:32.566380 26948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:24:32.566561 26948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "a6e3d5d709534a27bc15b2f371adbfb8"
format_stamp: "Formatted at 2026-08-09 06:24:32 on dist-test-slave-2w3w"
server_key: "e78c80344f2cebe107cae5faf18dcaf6"
server_key_iv: "edeb86ebe3473eda0561b2d74ecf2500"
server_key_version: "encryptionkey@0"
I20260809 06:24:32.566758 26948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:32.572184 26948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:32.575999 26948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:32.576301 26948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:32.594895 26948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40049
I20260809 06:24:32.594920 27018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40049 every 8 connection(s)
I20260809 06:24:32.596354 26948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:32.599997 27019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:32.602672 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 26948
I20260809 06:24:32.602921 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:32.603407 3621 external_mini_cluster.cc:1428] Setting key cda6aa1e6506c1cb2de0cfd0dba7e0dc
I20260809 06:24:32.611140 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:32.605311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:45697 (local address 127.0.0.1:40049)
0809 06:24:32.606064 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:32.606075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:32.606115 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:24:32.607822 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:32.607844 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:32.608170 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:32.608560 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:32.609597 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:32.609602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:32.609614 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:32.609631 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:32.609681 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:32.609703 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:32.609923 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:32.610059 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 06:24:32.610238 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":322,"threads_started":1}
I20260809 06:24:32.611490 27019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8: Bootstrap starting.
I20260809 06:24:32.614753 27019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:32.614769 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:32.615811 27019 log.cc:824] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:32.618659 27019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8: No bootstrap required, opened a new log
I20260809 06:24:32.622144 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:32.619739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:56857 (local address 127.0.0.1:40049)
0809 06:24:32.619929 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:32.619934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:32.620270 (+ 336us) server_negotiation.cc:407] Connection header received
0809 06:24:32.620421 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:32.620425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:32.620471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:32.620551 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:32.621481 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:32.621485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:32.621487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:32.621501 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:32.621535 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:32.621551 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:32.621638 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:32.621781 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 06:24:32.621827 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:24:32.626914 27019 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } }
I20260809 06:24:32.627348 27019 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:32.627460 27019 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3d5d709534a27bc15b2f371adbfb8, State: Initialized, Role: FOLLOWER
I20260809 06:24:32.628077 27019 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } }
I20260809 06:24:32.628332 27019 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:32.628485 27019 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:32.628677 27019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:32.631186 27019 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } }
I20260809 06:24:32.631631 27019 leader_election.cc:302] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e3d5d709534a27bc15b2f371adbfb8; no voters:
I20260809 06:24:32.632139 27019 leader_election.cc:288] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:32.632302 27031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:32.632961 27031 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [term 1 LEADER]: Becoming Leader. State: Replica: a6e3d5d709534a27bc15b2f371adbfb8, State: Running, Role: LEADER
I20260809 06:24:32.633459 27031 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } }
I20260809 06:24:32.634277 27019 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:32.636582 27034 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e3d5d709534a27bc15b2f371adbfb8. Latest consensus state: current_term: 1 leader_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } } }
I20260809 06:24:32.637039 27034 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:32.636554 27032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e3d5d709534a27bc15b2f371adbfb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40049 } } }
I20260809 06:24:32.637718 27032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:32.638711 27041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:32.644174 27041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:32.652937 27041 catalog_manager.cc:1347] Generated new cluster ID: 94e81843153a4dccb5896186406be621
I20260809 06:24:32.653059 27041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:32.696657 27041 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:32.697103 27041 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:32.705499 27041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e3d5d709534a27bc15b2f371adbfb8: Generated new TSK 0
I20260809 06:24:32.706274 27041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:32.721913 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26948
W20260809 06:24:32.723017 27013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:45697
W20260809 06:24:32.723805 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 26948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:32.723971 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 26948
2026-08-09T06:24:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:32 dist-test-slave-2w3w krb5kdc[27054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:32 dist-test-slave-2w3w krb5kdc[27054](info): set up 2 sockets
Aug 09 06:24:32 dist-test-slave-2w3w krb5kdc[27054](info): commencing operation
krb5kdc: starting...
W20260809 06:24:33.909034 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:34.209894 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46865
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:34.541258 27069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:34.541635 27069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:34.541806 27069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:34.548862 27069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:34.549005 27069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:34.549104 27069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:34.549207 27069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:34.549289 27069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:34.558936 27069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:46865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:34.560122 27069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:34.561429 27069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:34.573390 27077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:34.573441 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:34.573751 27074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:34.575013 27069 server_base.cc:1034] running on GCE node
I20260809 06:24:34.575769 27069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:34.577140 27069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:34.578369 27069 hybrid_clock.cc:648] HybridClock initialized: now 1786256674578339 us; error 54 us; skew 500 ppm
I20260809 06:24:34.578715 27069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:34.581771 27069 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:34.583223 27069 webserver.cc:460] Webserver started at http://127.3.137.126:39585/ using document root <none> and password file <none>
I20260809 06:24:34.583864 27069 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:34.584022 27069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:34.584326 27069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:34.587328 27069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "16ee8e798944462582b0a423cc5fda5d"
format_stamp: "Formatted at 2026-08-09 06:24:34 on dist-test-slave-2w3w"
server_key: "7fb0c502288e77b65e636ebb1148d446"
server_key_iv: "81aa36b4371afa94f9ea8b06e7f073a9"
server_key_version: "encryptionkey@0"
I20260809 06:24:34.588109 27069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "16ee8e798944462582b0a423cc5fda5d"
format_stamp: "Formatted at 2026-08-09 06:24:34 on dist-test-slave-2w3w"
server_key: "7fb0c502288e77b65e636ebb1148d446"
server_key_iv: "81aa36b4371afa94f9ea8b06e7f073a9"
server_key_version: "encryptionkey@0"
I20260809 06:24:34.593499 27069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 06:24:34.596987 27085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:34.598307 27069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:24:34.598491 27069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "16ee8e798944462582b0a423cc5fda5d"
format_stamp: "Formatted at 2026-08-09 06:24:34 on dist-test-slave-2w3w"
server_key: "7fb0c502288e77b65e636ebb1148d446"
server_key_iv: "81aa36b4371afa94f9ea8b06e7f073a9"
server_key_version: "encryptionkey@0"
I20260809 06:24:34.598694 27069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:34.609071 27069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:34.612874 27069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:34.613168 27069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:34.631567 27069 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:46865
I20260809 06:24:34.631600 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:46865 every 8 connection(s)
I20260809 06:24:34.632916 27069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:34.636511 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:34.639299 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27069
I20260809 06:24:34.639494 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:34.639925 3621 external_mini_cluster.cc:1428] Setting key 559aef2802a45d9c744944913b62fe6c
I20260809 06:24:34.648512 27142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d: Bootstrap starting.
I20260809 06:24:34.652138 27142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:34.653409 27142 log.cc:824] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:34.656237 27142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d: No bootstrap required, opened a new log
I20260809 06:24:34.664007 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:34.641652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40711 (local address 127.3.137.126:46865)
0809 06:24:34.642461 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:34.642472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:34.642511 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:24:34.644195 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:34.644216 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:34.644578 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:34.644875 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:34.646598 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:34.648055 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:34.649718 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:34.650342 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:34.650476 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:34.655546 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:34.655554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:34.655566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:34.655598 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:34.659635 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:34.660374 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:34.660383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:34.660395 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:34.660589 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:34.661032 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:34.661037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:34.661041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:34.661711 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:34.662062 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:34.662382 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 06:24:34.662711 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":414,"threads_started":1}
I20260809 06:24:34.666762 27142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } }
I20260809 06:24:34.667263 27142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:34.667404 27142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ee8e798944462582b0a423cc5fda5d, State: Initialized, Role: FOLLOWER
I20260809 06:24:34.668013 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:34.668064 27142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } }
I20260809 06:24:34.668352 27142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:34.668499 27142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:34.668666 27142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:34.670804 27142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } }
I20260809 06:24:34.671360 27142 leader_election.cc:302] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ee8e798944462582b0a423cc5fda5d; no voters:
I20260809 06:24:34.672004 27142 leader_election.cc:288] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:34.672101 27152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:34.676380 27152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [term 1 LEADER]: Becoming Leader. State: Replica: 16ee8e798944462582b0a423cc5fda5d, State: Running, Role: LEADER
I20260809 06:24:34.677099 27142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:34.677207 27152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } }
I20260809 06:24:34.680354 27153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ee8e798944462582b0a423cc5fda5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } } }
I20260809 06:24:34.680788 27153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:34.683207 27159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:34.682900 27154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ee8e798944462582b0a423cc5fda5d. Latest consensus state: current_term: 1 leader_uuid: "16ee8e798944462582b0a423cc5fda5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee8e798944462582b0a423cc5fda5d" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 46865 } } }
I20260809 06:24:34.683447 27154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:34.688761 27159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:34.694648 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:34.672592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57903 (local address 127.3.137.126:46865)
0809 06:24:34.672788 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:34.672797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:34.676905 (+ 4108us) server_negotiation.cc:407] Connection header received
0809 06:24:34.677219 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:34.677224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:34.677301 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:34.677417 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:34.679134 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:34.680327 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:34.682022 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:34.682604 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:34.682717 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:34.686463 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:34.686470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:34.686475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:34.686510 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:34.690854 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:34.691900 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:34.691906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:34.691910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:34.692058 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:34.692692 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:34.692697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:34.692700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:34.693149 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:34.693416 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:34.694090 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 06:24:34.694273 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 06:24:34.700548 27159 catalog_manager.cc:1347] Generated new cluster ID: 9a7bbfe471994e0986e25fc9212b5737
I20260809 06:24:34.700654 27159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:34.720124 27159 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:34.721526 27159 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:34.734660 27159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ee8e798944462582b0a423cc5fda5d: Generated new TSK 0
I20260809 06:24:34.735576 27159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:34.743402 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27069
I20260809 06:24:34.744557 27137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:34.833474 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27069
W20260809 06:24:34.853091 27052 connection.cc:535] client connection to 127.3.137.126:46865 recv error: Network error: failed to read from TLS socket (remote: 127.3.137.126:46865): Connection reset by peer (error 104)
2026-08-09T06:24:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27177](info): set up 2 sockets
Aug 09 06:24:34 dist-test-slave-2w3w krb5kdc[27177](info): commencing operation
krb5kdc: starting...
W20260809 06:24:35.996276 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:36 dist-test-slave-2w3w krb5kdc[27177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:36.298457 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33705
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:33705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:36.644505 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:36.644817 27191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:36.644985 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:36.652351 27191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:36.652477 27191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:36.652580 27191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:36.652679 27191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:36.652742 27191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:36.662214 27191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:33705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:33705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:36.663516 27191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:36.664768 27191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:36.676476 27196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:36.676422 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:36.676378 27199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:36.677119 27191 server_base.cc:1034] running on GCE node
I20260809 06:24:36.677946 27191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:36.679435 27191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:36.680671 27191 hybrid_clock.cc:648] HybridClock initialized: now 1786256676680625 us; error 41 us; skew 500 ppm
I20260809 06:24:36.681031 27191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:36 dist-test-slave-2w3w krb5kdc[27177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256676, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:36.684229 27191 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:36.685760 27191 webserver.cc:460] Webserver started at http://127.3.137.126:33113/ using document root <none> and password file <none>
I20260809 06:24:36.686450 27191 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:36.686602 27191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:36.686908 27191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:36.689927 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "15406e1055e646f592e4a724b9693cdc"
format_stamp: "Formatted at 2026-08-09 06:24:36 on dist-test-slave-2w3w"
server_key: "348f146335a5255713441de7d6228bee"
server_key_iv: "4e7b25364d536c65d3f3ca58edc465f3"
server_key_version: "encryptionkey@0"
I20260809 06:24:36.690737 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "15406e1055e646f592e4a724b9693cdc"
format_stamp: "Formatted at 2026-08-09 06:24:36 on dist-test-slave-2w3w"
server_key: "348f146335a5255713441de7d6228bee"
server_key_iv: "4e7b25364d536c65d3f3ca58edc465f3"
server_key_version: "encryptionkey@0"
I20260809 06:24:36.695931 27191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:24:36.699414 27207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:36.700754 27191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:24:36.701054 27191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "15406e1055e646f592e4a724b9693cdc"
format_stamp: "Formatted at 2026-08-09 06:24:36 on dist-test-slave-2w3w"
server_key: "348f146335a5255713441de7d6228bee"
server_key_iv: "4e7b25364d536c65d3f3ca58edc465f3"
server_key_version: "encryptionkey@0"
I20260809 06:24:36.701262 27191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:36.718173 27191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:36.722353 27191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:36.722651 27191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:36.741544 27191 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:33705
I20260809 06:24:36.741593 27262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:33705 every 8 connection(s)
I20260809 06:24:36.742933 27191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:36.746708 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:36.750556 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27191
I20260809 06:24:36.750752 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:36.751271 3621 external_mini_cluster.cc:1428] Setting key 1ea53e491f8f0f7d396e37cdfc08a1c4
I20260809 06:24:36.759331 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc: Bootstrap starting.
I20260809 06:24:36.762869 27263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:36.764009 27263 log.cc:824] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:36.764366 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:36.753061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33631 (local address 127.3.137.126:33705)
0809 06:24:36.753817 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:36.753828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:36.753866 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:24:36.755514 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:36.755535 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:36.755924 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:36.756227 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:36.756818 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:36.758143 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:36.759960 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:36.760601 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:36.760728 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:36.762120 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:36.762126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:36.762138 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:36.762623 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:36.762688 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:36.762711 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:36.762831 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:36.763198 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 06:24:36.763377 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":371,"threads_started":1}
I20260809 06:24:36.766778 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc: No bootstrap required, opened a new log
I20260809 06:24:36.768365 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:36.774905 27263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } }
I20260809 06:24:36.775364 27263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:36.775532 27263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15406e1055e646f592e4a724b9693cdc, State: Initialized, Role: FOLLOWER
I20260809 06:24:36.776160 27263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } }
I20260809 06:24:36.776429 27263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:36.776568 27263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:36.776738 27263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:36.779609 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:36.772811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55317 (local address 127.3.137.126:33705)
0809 06:24:36.773031 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:36.773037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:36.773366 (+ 329us) server_negotiation.cc:407] Connection header received
0809 06:24:36.773528 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:36.773532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:36.773588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:36.773672 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:36.774246 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:36.775061 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:36.776994 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:36.777426 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:36.777527 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:36.778781 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:36.778785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:36.778788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:36.778804 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:36.778855 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:36.778870 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:36.778951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:36.779212 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 06:24:36.779276 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:24:36.779860 27263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } }
I20260809 06:24:36.780385 27263 leader_election.cc:302] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15406e1055e646f592e4a724b9693cdc; no voters:
I20260809 06:24:36.781029 27263 leader_election.cc:288] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:36.781148 27273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:36.781840 27273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [term 1 LEADER]: Becoming Leader. State: Replica: 15406e1055e646f592e4a724b9693cdc, State: Running, Role: LEADER
I20260809 06:24:36.782418 27273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } }
I20260809 06:24:36.783198 27263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:36.785320 27274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15406e1055e646f592e4a724b9693cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } } }
I20260809 06:24:36.785684 27274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:36.786155 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15406e1055e646f592e4a724b9693cdc. Latest consensus state: current_term: 1 leader_uuid: "15406e1055e646f592e4a724b9693cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15406e1055e646f592e4a724b9693cdc" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 33705 } } }
I20260809 06:24:36.786549 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:36.789301 27280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:36.794409 27280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:36.805734 27280 catalog_manager.cc:1347] Generated new cluster ID: b590200dc89843ffbd41c79405ea26a4
I20260809 06:24:36.805838 27280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:36.814989 27280 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:36.816183 27280 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:36.823886 27280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15406e1055e646f592e4a724b9693cdc: Generated new TSK 0
I20260809 06:24:36.824628 27280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:36.830022 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27191
W20260809 06:24:36.831202 27258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33631
W20260809 06:24:36.832197 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27191: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:36.832370 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27191
W20260809 06:24:36.852339 27175 connection.cc:535] client connection to 127.3.137.126:33705 recv error: Network error: failed to read from TLS socket (remote: 127.3.137.126:33705): Connection reset by peer (error 104)
2026-08-09T06:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:36 dist-test-slave-2w3w krb5kdc[27296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:36 dist-test-slave-2w3w krb5kdc[27296](info): set up 2 sockets
Aug 09 06:24:36 dist-test-slave-2w3w krb5kdc[27296](info): commencing operation
krb5kdc: starting...
W20260809 06:24:37.989773 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:38 dist-test-slave-2w3w krb5kdc[27296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:38.293844 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:40033
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:38.625442 27309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:38.625735 27309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:38.625828 27309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:38.633004 27309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:38.633117 27309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:38.633163 27309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:38.633232 27309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:38.633277 27309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:38.643014 27309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:40033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:38.644222 27309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:38.645519 27309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:38.656971 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:38.657047 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:38.657019 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:38.658057 27309 server_base.cc:1034] running on GCE node
I20260809 06:24:38.658737 27309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:38.660120 27309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:38.661316 27309 hybrid_clock.cc:648] HybridClock initialized: now 1786256678661285 us; error 30 us; skew 500 ppm
I20260809 06:24:38.661662 27309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:38 dist-test-slave-2w3w krb5kdc[27296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:38.668506 27309 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:38.670033 27309 webserver.cc:460] Webserver started at http://127.3.137.126:45715/ using document root <none> and password file <none>
I20260809 06:24:38.670694 27309 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:38.670848 27309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:38.671221 27309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:38.674124 27309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "97c9b93341c448cca2a8333199f82eb6"
format_stamp: "Formatted at 2026-08-09 06:24:38 on dist-test-slave-2w3w"
server_key: "5d9452233cbe6bf2a572e8b2cf31c60a"
server_key_iv: "8141ee862d41e2e53a6cd613aba8f926"
server_key_version: "encryptionkey@0"
I20260809 06:24:38.674901 27309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "97c9b93341c448cca2a8333199f82eb6"
format_stamp: "Formatted at 2026-08-09 06:24:38 on dist-test-slave-2w3w"
server_key: "5d9452233cbe6bf2a572e8b2cf31c60a"
server_key_iv: "8141ee862d41e2e53a6cd613aba8f926"
server_key_version: "encryptionkey@0"
I20260809 06:24:38.680130 27309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:24:38.683458 27325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:38.684727 27309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:24:38.684911 27309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "97c9b93341c448cca2a8333199f82eb6"
format_stamp: "Formatted at 2026-08-09 06:24:38 on dist-test-slave-2w3w"
server_key: "5d9452233cbe6bf2a572e8b2cf31c60a"
server_key_iv: "8141ee862d41e2e53a6cd613aba8f926"
server_key_version: "encryptionkey@0"
I20260809 06:24:38.685114 27309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:38.690775 27309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:38.694689 27309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:38.695072 27309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:38.713732 27309 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:40033
I20260809 06:24:38.713793 27377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:40033 every 8 connection(s)
I20260809 06:24:38.715144 27309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:38.718657 27378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:38.722293 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27309
I20260809 06:24:38.722481 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:38.723050 3621 external_mini_cluster.cc:1428] Setting key 77be7809169441d88f58c298e51bec20
I20260809 06:24:38.730515 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:38.724784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33515 (local address 127.3.137.126:40033)
0809 06:24:38.725578 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:38.725589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:38.725627 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:24:38.727283 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:38.727305 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:38.727637 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:38.727951 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:38.729032 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:38.729038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:38.729049 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:38.729069 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:38.729128 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:38.729149 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:38.729312 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:38.729469 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 06:24:38.729651 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260809 06:24:38.731413 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6: Bootstrap starting.
I20260809 06:24:38.734097 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:38.734701 27378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:38.735751 27378 log.cc:824] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:38.738994 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6: No bootstrap required, opened a new log
I20260809 06:24:38.741184 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:38.738499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52595 (local address 127.3.137.126:40033)
0809 06:24:38.738693 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:38.738701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:38.739272 (+ 571us) server_negotiation.cc:407] Connection header received
0809 06:24:38.739438 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:38.739441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:38.739486 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:38.739560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:38.740510 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:38.740514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:38.740517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:38.740530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:38.740566 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:38.740580 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:38.740663 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:38.740825 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 06:24:38.740873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:24:38.746757 27378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } }
I20260809 06:24:38.747174 27378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:38.747327 27378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c9b93341c448cca2a8333199f82eb6, State: Initialized, Role: FOLLOWER
I20260809 06:24:38.747961 27378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } }
I20260809 06:24:38.748206 27378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:38.748358 27378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:38.748528 27378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:38.750694 27378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } }
I20260809 06:24:38.751228 27378 leader_election.cc:302] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c9b93341c448cca2a8333199f82eb6; no voters:
I20260809 06:24:38.751866 27378 leader_election.cc:288] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:38.752012 27389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:38.752763 27389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 97c9b93341c448cca2a8333199f82eb6, State: Running, Role: LEADER
I20260809 06:24:38.753309 27389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } }
I20260809 06:24:38.753908 27378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:38.756129 27391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c9b93341c448cca2a8333199f82eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } } }
I20260809 06:24:38.756553 27391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:38.756610 27392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c9b93341c448cca2a8333199f82eb6. Latest consensus state: current_term: 1 leader_uuid: "97c9b93341c448cca2a8333199f82eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9b93341c448cca2a8333199f82eb6" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 40033 } } }
I20260809 06:24:38.756898 27392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:38.758360 27397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:38.763756 27397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:38.771962 27397 catalog_manager.cc:1347] Generated new cluster ID: fd7dfc2dcbb749fd97f363046fe8c00c
I20260809 06:24:38.772070 27397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:38.813410 27397 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:38.813860 27397 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:38.825295 27397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c9b93341c448cca2a8333199f82eb6: Generated new TSK 0
I20260809 06:24:38.826121 27397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:38.832861 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27309
W20260809 06:24:38.834242 27373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33515
W20260809 06:24:38.834952 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:38.835130 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27309
2026-08-09T06:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:38 dist-test-slave-2w3w krb5kdc[27412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:38 dist-test-slave-2w3w krb5kdc[27412](info): set up 2 sockets
Aug 09 06:24:38 dist-test-slave-2w3w krb5kdc[27412](info): commencing operation
krb5kdc: starting...
W20260809 06:24:40.001230 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:40 dist-test-slave-2w3w krb5kdc[27412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:40.306875 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35525
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:40.637270 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:40.637557 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:40.637650 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:40.644706 27425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:40.644835 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:40.644940 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:40.645044 27425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:40.645128 27425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:40.654568 27425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:40.655826 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:40.657054 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:40.668879 27432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:40.668915 27435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:40.668941 27433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:40.670110 27425 server_base.cc:1034] running on GCE node
I20260809 06:24:40.670764 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:40.672122 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:40.673337 27425 hybrid_clock.cc:648] HybridClock initialized: now 1786256680673276 us; error 45 us; skew 500 ppm
I20260809 06:24:40.673671 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:40 dist-test-slave-2w3w krb5kdc[27412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256680, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:40.676774 27425 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:40.678275 27425 webserver.cc:460] Webserver started at http://127.3.137.126:36751/ using document root <none> and password file <none>
I20260809 06:24:40.678934 27425 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:40.679147 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:40.679450 27425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:40.682426 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "c6e7b093d93e4368a9577990f86b57c8"
format_stamp: "Formatted at 2026-08-09 06:24:40 on dist-test-slave-2w3w"
server_key: "68a19692b47974d4c4ffc97fb221b0c8"
server_key_iv: "96d793d5527de69e680ff29beffeb22d"
server_key_version: "encryptionkey@0"
I20260809 06:24:40.683277 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "c6e7b093d93e4368a9577990f86b57c8"
format_stamp: "Formatted at 2026-08-09 06:24:40 on dist-test-slave-2w3w"
server_key: "68a19692b47974d4c4ffc97fb221b0c8"
server_key_iv: "96d793d5527de69e680ff29beffeb22d"
server_key_version: "encryptionkey@0"
I20260809 06:24:40.688205 27425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:24:40.691533 27442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:40.692679 27425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:24:40.692916 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "c6e7b093d93e4368a9577990f86b57c8"
format_stamp: "Formatted at 2026-08-09 06:24:40 on dist-test-slave-2w3w"
server_key: "68a19692b47974d4c4ffc97fb221b0c8"
server_key_iv: "96d793d5527de69e680ff29beffeb22d"
server_key_version: "encryptionkey@0"
I20260809 06:24:40.693140 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:40.723142 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:40.726995 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:40.727285 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:40.750754 27425 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:35525
I20260809 06:24:40.750753 27499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:35525 every 8 connection(s)
I20260809 06:24:40.752589 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:40.756387 27501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:40.756536 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27425
I20260809 06:24:40.756736 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:40.757243 3621 external_mini_cluster.cc:1428] Setting key 428bbcb89e535efeeed5e355980b9ae2
I20260809 06:24:40.770874 27501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8: Bootstrap starting.
Aug 09 06:24:40 dist-test-slave-2w3w krb5kdc[27412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:40.774777 27501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:40.776216 27501 log.cc:824] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:40.779623 27501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8: No bootstrap required, opened a new log
I20260809 06:24:40.781548 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:40.759100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35565 (local address 127.3.137.126:35525)
0809 06:24:40.759924 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:40.759938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:40.759982 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:24:40.761649 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:40.761682 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:40.762176 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:40.762606 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:40.763981 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:40.765820 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:40.767235 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:40.768319 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:40.768517 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:40.772872 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:40.772883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:40.772901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:40.772951 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:40.777141 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:40.777959 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:40.777967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:40.777975 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:40.778178 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:40.778559 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:40.778564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:40.778567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:40.779327 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:40.779663 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:40.780142 (+ 479us) server_negotiation.cc:299] Negotiation successful
0809 06:24:40.780559 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
I20260809 06:24:40.785293 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:40.787700 27501 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } }
I20260809 06:24:40.788117 27501 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:40.788237 27501 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e7b093d93e4368a9577990f86b57c8, State: Initialized, Role: FOLLOWER
I20260809 06:24:40.788841 27501 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } }
I20260809 06:24:40.789096 27501 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:40.789245 27501 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:40.789433 27501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:40.792062 27501 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } }
I20260809 06:24:40.792574 27501 leader_election.cc:302] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6e7b093d93e4368a9577990f86b57c8; no voters:
I20260809 06:24:40.793264 27501 leader_election.cc:288] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:40.793398 27511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:40.795912 27501 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:40.799240 27511 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [term 1 LEADER]: Becoming Leader. State: Replica: c6e7b093d93e4368a9577990f86b57c8, State: Running, Role: LEADER
I20260809 06:24:40.799839 27511 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } }
I20260809 06:24:40.803261 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6e7b093d93e4368a9577990f86b57c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } } }
I20260809 06:24:40.803654 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:40.804159 27513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6e7b093d93e4368a9577990f86b57c8. Latest consensus state: current_term: 1 leader_uuid: "c6e7b093d93e4368a9577990f86b57c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e7b093d93e4368a9577990f86b57c8" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 35525 } } }
I20260809 06:24:40.804466 27513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:40.808064 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:40.789809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40741 (local address 127.3.137.126:35525)
0809 06:24:40.789989 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:40.789994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:40.790481 (+ 487us) server_negotiation.cc:407] Connection header received
0809 06:24:40.790588 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:40.790594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:40.790679 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:40.790786 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:40.792296 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:40.793124 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:40.794858 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:40.795443 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:40.795608 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:40.799032 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:40.799045 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:40.799051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:40.799089 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:40.803043 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:40.804082 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:40.804089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:40.804094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:40.804263 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:40.804774 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:40.804779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:40.804782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:40.805287 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:40.807091 (+ 1804us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:40.807260 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 06:24:40.807538 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:24:40.817847 27522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:40.823361 27522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:40.832804 27522 catalog_manager.cc:1347] Generated new cluster ID: 14de3fff217b465ea1f97df60983c42d
I20260809 06:24:40.832923 27522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:40.846050 27522 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:40.847878 27522 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:40.865027 27522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6e7b093d93e4368a9577990f86b57c8: Generated new TSK 0
I20260809 06:24:40.865789 27522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:40.873320 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27425
I20260809 06:24:40.874606 27494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:40.966303 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27425
2026-08-09T06:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2130 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:41 dist-test-slave-2w3w krb5kdc[27537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:41 dist-test-slave-2w3w krb5kdc[27537](info): set up 2 sockets
Aug 09 06:24:41 dist-test-slave-2w3w krb5kdc[27537](info): commencing operation
krb5kdc: starting...
W20260809 06:24:42.129281 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:42 dist-test-slave-2w3w krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:42.458441 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:36707
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:36707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:42.803818 27550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:42.804157 27550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:42.804291 27550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:42.812120 27550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:42.812230 27550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:42.812276 27550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:42.812350 27550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:42.812395 27550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:42.822193 27550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:36707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:42.823436 27550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:42.824710 27550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:42.836670 27555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:42.836716 27558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:42.836616 27556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:42.837553 27550 server_base.cc:1034] running on GCE node
I20260809 06:24:42.838346 27550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:42.839756 27550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:42.840996 27550 hybrid_clock.cc:648] HybridClock initialized: now 1786256682840962 us; error 48 us; skew 500 ppm
I20260809 06:24:42.841334 27550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:42 dist-test-slave-2w3w krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256682, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:42.844481 27550 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:42.845793 27550 webserver.cc:460] Webserver started at http://127.3.137.126:34569/ using document root <none> and password file <none>
I20260809 06:24:42.846398 27550 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:42.846499 27550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:42.846742 27550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:42.849951 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "c40e08ee3fb94dd38a70967c67b01292"
format_stamp: "Formatted at 2026-08-09 06:24:42 on dist-test-slave-2w3w"
server_key: "d10ce35eb056ec6855bb05c40a363bb1"
server_key_iv: "1e66761490a2837e91298fc6afff92f0"
server_key_version: "encryptionkey@0"
I20260809 06:24:42.850777 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "c40e08ee3fb94dd38a70967c67b01292"
format_stamp: "Formatted at 2026-08-09 06:24:42 on dist-test-slave-2w3w"
server_key: "d10ce35eb056ec6855bb05c40a363bb1"
server_key_iv: "1e66761490a2837e91298fc6afff92f0"
server_key_version: "encryptionkey@0"
I20260809 06:24:42.855827 27550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 06:24:42.859189 27565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:42.860566 27550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:24:42.860747 27550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "c40e08ee3fb94dd38a70967c67b01292"
format_stamp: "Formatted at 2026-08-09 06:24:42 on dist-test-slave-2w3w"
server_key: "d10ce35eb056ec6855bb05c40a363bb1"
server_key_iv: "1e66761490a2837e91298fc6afff92f0"
server_key_version: "encryptionkey@0"
I20260809 06:24:42.860925 27550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:42.890067 27550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:42.894071 27550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:42.894363 27550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:42.913093 27550 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:36707
I20260809 06:24:42.913128 27620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:36707 every 8 connection(s)
I20260809 06:24:42.914434 27550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:42.918047 27621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:42.921643 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27550
I20260809 06:24:42.921828 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:42.922410 3621 external_mini_cluster.cc:1428] Setting key fb26c9749a7cc6427f912fee201c119b
I20260809 06:24:42.930429 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292: Bootstrap starting.
I20260809 06:24:42.934180 27621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:42.935375 27621 log.cc:824] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:42.935736 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:42.924181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56117 (local address 127.3.137.126:36707)
0809 06:24:42.924923 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:42.924935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:42.924975 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:24:42.926594 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:42.926615 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:42.927064 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:42.927385 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:42.927978 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:42.929328 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:42.931188 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:42.931830 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:42.931970 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:42.933335 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:42.933342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:42.933354 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:42.933870 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:42.933947 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:42.933971 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:42.934055 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:42.934362 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 06:24:42.934671 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
I20260809 06:24:42.938531 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292: No bootstrap required, opened a new log
I20260809 06:24:42.939435 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:42.949277 27621 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } }
I20260809 06:24:42.949779 27621 raft_consensus.cc:374] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:42.949896 27621 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40e08ee3fb94dd38a70967c67b01292, State: Initialized, Role: FOLLOWER
I20260809 06:24:42.950641 27621 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } }
I20260809 06:24:42.951014 27621 raft_consensus.cc:388] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:42.951386 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:42.944696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41805 (local address 127.3.137.126:36707)
0809 06:24:42.944931 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:42.944936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:42.945281 (+ 345us) server_negotiation.cc:407] Connection header received
0809 06:24:42.945479 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:42.945482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:42.945544 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:42.945622 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:42.946210 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:42.946933 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:42.948862 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:42.949274 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:42.949363 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:42.950558 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:42.950562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:42.950565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:42.950583 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:42.950618 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:42.950636 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:42.950706 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:42.950917 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 06:24:42.951046 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 06:24:42.952313 27621 raft_consensus.cc:482] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:42.952550 27621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:42.955384 27621 raft_consensus.cc:504] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } }
I20260809 06:24:42.955972 27621 leader_election.cc:302] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c40e08ee3fb94dd38a70967c67b01292; no voters:
I20260809 06:24:42.956557 27621 leader_election.cc:288] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:42.956683 27632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:42.957403 27632 raft_consensus.cc:686] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [term 1 LEADER]: Becoming Leader. State: Replica: c40e08ee3fb94dd38a70967c67b01292, State: Running, Role: LEADER
I20260809 06:24:42.957926 27632 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } }
I20260809 06:24:42.958911 27621 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:42.961416 27634 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40e08ee3fb94dd38a70967c67b01292. Latest consensus state: current_term: 1 leader_uuid: "c40e08ee3fb94dd38a70967c67b01292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } } }
I20260809 06:24:42.961385 27633 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40e08ee3fb94dd38a70967c67b01292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e08ee3fb94dd38a70967c67b01292" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 36707 } } }
I20260809 06:24:42.961926 27634 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:42.961959 27633 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:42.963667 27640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:42.968922 27640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:42.978085 27640 catalog_manager.cc:1347] Generated new cluster ID: ee23f21de35941de9c6ea4b250d7a056
I20260809 06:24:42.978232 27640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:43.006778 27640 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:43.008772 27640 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:43.024317 27640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c40e08ee3fb94dd38a70967c67b01292: Generated new TSK 0
I20260809 06:24:43.025434 27640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:43.041760 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27550
W20260809 06:24:43.042948 27616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56117
W20260809 06:24:43.043757 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:43.043915 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27550
2026-08-09T06:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2079 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:43 dist-test-slave-2w3w krb5kdc[27654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:43 dist-test-slave-2w3w krb5kdc[27654](info): set up 2 sockets
Aug 09 06:24:43 dist-test-slave-2w3w krb5kdc[27654](info): commencing operation
krb5kdc: starting...
W20260809 06:24:44.234187 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:44 dist-test-slave-2w3w krb5kdc[27654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:44.562282 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:41603
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:44.911166 27667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:44.911520 27667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:44.911687 27667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:44.919090 27667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:44.919246 27667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:44.919365 27667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:44.919477 27667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:44.919569 27667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:44.929515 27667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:44.930896 27667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:44.932233 27667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:44.944128 27673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:44.944299 27674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:44.944308 27676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:44.945549 27667 server_base.cc:1034] running on GCE node
I20260809 06:24:44.946376 27667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:44.947865 27667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:44.949111 27667 hybrid_clock.cc:648] HybridClock initialized: now 1786256684949055 us; error 46 us; skew 500 ppm
I20260809 06:24:44.949478 27667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:44 dist-test-slave-2w3w krb5kdc[27654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256684, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:44.957520 27667 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:44.959096 27667 webserver.cc:460] Webserver started at http://127.3.137.126:43763/ using document root <none> and password file <none>
I20260809 06:24:44.959811 27667 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:44.959969 27667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:44.960289 27667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:44.963479 27667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "27b315079bea4e21b84969a1f72ecca0"
format_stamp: "Formatted at 2026-08-09 06:24:44 on dist-test-slave-2w3w"
server_key: "03396deceb9d6602be2ff34a2328e48a"
server_key_iv: "c4bcd9fce81206a937f050d5aff76237"
server_key_version: "encryptionkey@0"
I20260809 06:24:44.964321 27667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "27b315079bea4e21b84969a1f72ecca0"
format_stamp: "Formatted at 2026-08-09 06:24:44 on dist-test-slave-2w3w"
server_key: "03396deceb9d6602be2ff34a2328e48a"
server_key_iv: "c4bcd9fce81206a937f050d5aff76237"
server_key_version: "encryptionkey@0"
I20260809 06:24:44.969620 27667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:24:44.973165 27684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:44.974486 27667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:24:44.974675 27667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "27b315079bea4e21b84969a1f72ecca0"
format_stamp: "Formatted at 2026-08-09 06:24:44 on dist-test-slave-2w3w"
server_key: "03396deceb9d6602be2ff34a2328e48a"
server_key_iv: "c4bcd9fce81206a937f050d5aff76237"
server_key_version: "encryptionkey@0"
I20260809 06:24:44.974896 27667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:44.980585 27667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:44.984731 27667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:44.985044 27667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:45.003553 27667 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:41603
I20260809 06:24:45.003598 27738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:41603 every 8 connection(s)
I20260809 06:24:45.005056 27667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:45.008941 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:45.014459 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27667
I20260809 06:24:45.014657 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:45.015214 3621 external_mini_cluster.cc:1428] Setting key 291347c6c1b74c289405d9600902cea0
I20260809 06:24:45.020279 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0: Bootstrap starting.
I20260809 06:24:45.023650 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:45.017226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40461 (local address 127.3.137.126:41603)
0809 06:24:45.018157 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:45.018172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:45.018224 (+ 52us) server_negotiation.cc:407] Connection header received
0809 06:24:45.020041 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:45.020065 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:45.020456 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:45.020849 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:45.022018 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:45.022025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:45.022040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:45.022055 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:45.022120 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:45.022144 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:45.022330 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:45.022442 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 06:24:45.022649 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":428,"threads_started":1}
I20260809 06:24:45.024330 27739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:45.025547 27739 log.cc:824] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:45.027868 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:45.028657 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0: No bootstrap required, opened a new log
I20260809 06:24:45.035017 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:45.032328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42075 (local address 127.3.137.126:41603)
0809 06:24:45.032578 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:45.032585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:45.032964 (+ 379us) server_negotiation.cc:407] Connection header received
0809 06:24:45.033138 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:45.033142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:45.033191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:45.033291 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:45.034278 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:45.034282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:45.034285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:24:45.034299 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:45.034340 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:24:45.034355 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:45.034458 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:45.034588 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 06:24:45.034634 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:24:45.037811 27739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } }
I20260809 06:24:45.038192 27739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:24:45.038326 27739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b315079bea4e21b84969a1f72ecca0, State: Initialized, Role: FOLLOWER
I20260809 06:24:45.039043 27739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } }
I20260809 06:24:45.039322 27739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:24:45.039482 27739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:24:45.039672 27739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:45.041908 27739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } }
I20260809 06:24:45.042526 27739 leader_election.cc:302] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27b315079bea4e21b84969a1f72ecca0; no voters:
I20260809 06:24:45.043265 27739 leader_election.cc:288] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:24:45.043383 27750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:45.044040 27750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [term 1 LEADER]: Becoming Leader. State: Replica: 27b315079bea4e21b84969a1f72ecca0, State: Running, Role: LEADER
I20260809 06:24:45.044623 27750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } }
I20260809 06:24:45.045745 27739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:45.047640 27751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27b315079bea4e21b84969a1f72ecca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } } }
I20260809 06:24:45.047748 27753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b315079bea4e21b84969a1f72ecca0. Latest consensus state: current_term: 1 leader_uuid: "27b315079bea4e21b84969a1f72ecca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b315079bea4e21b84969a1f72ecca0" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 41603 } } }
I20260809 06:24:45.048173 27751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:45.048261 27753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:45.056618 27755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:45.063045 27755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:45.072716 27755 catalog_manager.cc:1347] Generated new cluster ID: 8e20fc45465745c894fdf6c3d6e58d74
I20260809 06:24:45.072860 27755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:45.083793 27755 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:45.084251 27755 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:45.096539 27755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27b315079bea4e21b84969a1f72ecca0: Generated new TSK 0
I20260809 06:24:45.097455 27755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:24:45.113972 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27667
W20260809 06:24:45.115223 27734 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40461
W20260809 06:24:45.116076 3621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:24:45.116255 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27667
2026-08-09T06:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2071 ms)
[----------] 18 tests from AuthTokenIssuingTest (36912 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:45 dist-test-slave-2w3w krb5kdc[27773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:45 dist-test-slave-2w3w krb5kdc[27773](info): set up 2 sockets
Aug 09 06:24:45 dist-test-slave-2w3w krb5kdc[27773](info): commencing operation
krb5kdc: starting...
W20260809 06:24:46.312902 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:46 dist-test-slave-2w3w krb5kdc[27773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:46.675024 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:44511
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:47.025833 27787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:47.026216 27787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:47.026284 27787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:47.026369 27787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:47.034107 27787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:47.034233 27787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:47.034299 27787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:47.034348 27787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:47.034418 27787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:47.045785 27787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:47.047953 27787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:47.049734 27787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:47.063553 27793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:47.063582 27792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:47.063534 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:47.064388 27787 server_base.cc:1034] running on GCE node
I20260809 06:24:47.065230 27787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:47.066650 27787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:47.067871 27787 hybrid_clock.cc:648] HybridClock initialized: now 1786256687067784 us; error 59 us; skew 500 ppm
I20260809 06:24:47.068197 27787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:47.071481 27787 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:47.073022 27787 webserver.cc:460] Webserver started at http://127.3.137.126:34183/ using document root <none> and password file <none>
I20260809 06:24:47.073860 27787 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:47.073993 27787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:47.074358 27787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:47.077827 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "a3ef59dbb460449095a8c1fb44df06ef"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "3cd22acb5261456cfe2d3e8b7903b461"
server_key_iv: "c3b5f96534acce5269321122777d9cee"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.078673 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "a3ef59dbb460449095a8c1fb44df06ef"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "3cd22acb5261456cfe2d3e8b7903b461"
server_key_iv: "c3b5f96534acce5269321122777d9cee"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.084378 27787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:24:47.088100 27803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:47.089475 27787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:24:47.089665 27787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "a3ef59dbb460449095a8c1fb44df06ef"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "3cd22acb5261456cfe2d3e8b7903b461"
server_key_iv: "c3b5f96534acce5269321122777d9cee"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.089895 27787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:47.118577 27787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:47.123171 27787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:47.123495 27787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:47.143611 27787 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:44511
I20260809 06:24:47.143625 27855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:44511 every 8 connection(s)
I20260809 06:24:47.145018 27787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:47.148885 27856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:47.149351 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27787
I20260809 06:24:47.149574 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:47.150327 3621 external_mini_cluster.cc:1428] Setting key 16f800e1784b6f46d40714a153299e4b
I20260809 06:24:47.156136 27856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:47.183404 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.159671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44511 (local address 127.0.0.1:59384)
0809 06:24:47.160654 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.160718 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:47.163697 (+ 2979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:47.164811 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:47.164817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:47.165400 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:47.165789 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.165808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.167578 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.167583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.168761 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.168774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.169789 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.169797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.169979 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.170754 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:47.170779 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:47.173463 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:47.178268 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.178281 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.178292 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.178739 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.179307 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.179311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.179313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.179480 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.180602 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:47.180608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:47.180835 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 06:24:47.181117 (+ 282us) client_negotiation.cc:240] Negotiation successful
0809 06:24:47.181813 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":163,"thread_start_us":398,"threads_started":1}
I20260809 06:24:47.183423 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.159671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.3.137.126:44511)
0809 06:24:47.161687 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:47.161693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:47.161735 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:24:47.164150 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:47.164159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:47.164501 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:47.164746 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:47.166082 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.167410 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.168924 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.169534 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.169760 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.173748 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:47.173762 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:47.173795 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:47.173842 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:47.178091 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.178905 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.178918 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.178924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.179164 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.179614 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.179622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.179628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.180441 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:47.180661 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:47.181246 (+ 585us) server_negotiation.cc:299] Negotiation successful
0809 06:24:47.181640 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1807,"spinlock_wait_cycles":25984,"thread_start_us":1561,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.125@KRBTEST.COM; defaulting to no policy
I20260809 06:24:47.186522 27856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:47.187673 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.187259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37248)
0809 06:24:47.187432 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.187559 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.125@KRBTEST.COM" created.
W20260809 06:24:47.188094 27804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
I20260809 06:24:47.189879 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.189405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37260)
0809 06:24:47.189589 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.189697 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 06:24:47.190095 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:46501: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
I20260809 06:24:47.228127 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:46501
--webserver_interface=127.3.137.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:24:47.239492 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } attempt: 1
I20260809 06:24:47.240908 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.240304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37270)
0809 06:24:47.240569 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.240744 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 06:24:47.242054 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.241723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37272)
0809 06:24:47.241858 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.241929 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 06:24:47.242295 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:46501: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
I20260809 06:24:47.310568 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } attempt: 2
I20260809 06:24:47.311986 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.311454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37276)
0809 06:24:47.311715 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.311828 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 06:24:47.313126 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.312761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37292)
0809 06:24:47.312930 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.312998 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 06:24:47.313366 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:46501: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
I20260809 06:24:47.404609 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } attempt: 3
I20260809 06:24:47.405841 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.405396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37296)
0809 06:24:47.405608 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.405705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:24:47.406874 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.406539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37312)
0809 06:24:47.406669 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.406737 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:24:47.407105 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:46501: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
I20260809 06:24:47.550393 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } attempt: 4
I20260809 06:24:47.551857 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.551235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37324)
0809 06:24:47.551619 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.551713 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260809 06:24:47.553005 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.552603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37340)
0809 06:24:47.552763 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.552864 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:24:47.553201 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:46501: Network error: Client connection negotiation failed: client connection to 127.3.137.125:46501: connect: Connection refused (error 111)
W20260809 06:24:47.581081 27863 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:47.581502 27863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:47.581634 27863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:47.581787 27863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:47.589596 27863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:47.589761 27863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:47.589874 27863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:47.589969 27863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:47.590073 27863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:47.600836 27863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.137.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:47.602280 27863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:47.603698 27863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:47.616891 27871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:47.617473 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:47.618088 27873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:47.618152 27863 server_base.cc:1034] running on GCE node
I20260809 06:24:47.619115 27863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:47.620533 27863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:47.621788 27863 hybrid_clock.cc:648] HybridClock initialized: now 1786256687621749 us; error 58 us; skew 500 ppm
I20260809 06:24:47.622136 27863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:47.625430 27863 init.cc:377] Logged in from keytab as kudu/127.3.137.125@KRBTEST.COM (short username kudu)
I20260809 06:24:47.627203 27863 webserver.cc:460] Webserver started at http://127.3.137.125:39361/ using document root <none> and password file <none>
I20260809 06:24:47.627888 27863 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:47.627978 27863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:47.628261 27863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:47.631371 27863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data/instance:
uuid: "a114833b6bcb43db87a99047a02b5d44"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "02d407e6228f22b871881cbb83a24883"
server_key_iv: "d27b418f79b2c5fd9cd6eccbee15c286"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.632223 27863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal/instance:
uuid: "a114833b6bcb43db87a99047a02b5d44"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "02d407e6228f22b871881cbb83a24883"
server_key_iv: "d27b418f79b2c5fd9cd6eccbee15c286"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.637547 27863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:24:47.641291 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:47.642861 27863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:24:47.643132 27863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "a114833b6bcb43db87a99047a02b5d44"
format_stamp: "Formatted at 2026-08-09 06:24:47 on dist-test-slave-2w3w"
server_key: "02d407e6228f22b871881cbb83a24883"
server_key_iv: "d27b418f79b2c5fd9cd6eccbee15c286"
server_key_version: "encryptionkey@0"
I20260809 06:24:47.643357 27863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:47.661702 27863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:47.666079 27863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:47.666375 27863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:47.686591 27863 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.125:46501
I20260809 06:24:47.686612 27935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.125:46501 every 8 connection(s)
I20260809 06:24:47.688124 27863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
I20260809 06:24:47.692167 27937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:47.693393 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27863
I20260809 06:24:47.693578 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-1/wal/instance
I20260809 06:24:47.694123 3621 external_mini_cluster.cc:1428] Setting key 28fe2dcc08a508925ba23691a98862a9
I20260809 06:24:47.699314 27937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:47.722368 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.702093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.3.137.126:44511)
0809 06:24:47.702594 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:47.702599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:47.703726 (+ 1127us) server_negotiation.cc:407] Connection header received
0809 06:24:47.705687 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:47.705693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:47.705800 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:47.705935 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:47.707709 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.708444 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.710043 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.710474 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.710618 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.714737 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:47.714749 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:47.714757 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:47.714824 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:47.718649 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.719547 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.719556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.719563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.719721 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.720154 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.720162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.720167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.720598 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:47.720758 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:47.721580 (+ 822us) server_negotiation.cc:299] Negotiation successful
0809 06:24:47.721847 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":278,"threads_started":1}
I20260809 06:24:47.723093 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.702700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44511 (local address 127.0.0.1:59396)
0809 06:24:47.703548 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.703588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:47.705398 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:47.706193 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:47.706201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:47.706780 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:47.707470 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.707489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.708622 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.708627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.709883 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.709900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.710712 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.710719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.711047 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.711795 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:47.711818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:47.714425 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:47.718842 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.718849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.718863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.719395 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.719842 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.719846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.719848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.720028 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.720758 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:47.720764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:47.721151 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 06:24:47.721410 (+ 259us) client_negotiation.cc:240] Negotiation successful
0809 06:24:47.721833 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":364,"threads_started":1}
I20260809 06:24:47.725028 27937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.137.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.124@KRBTEST.COM" created.
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:24:47.741401 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.725884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37354)
0809 06:24:47.726081 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.726106 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:47.726249 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:47.727205 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:47.727209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:47.727611 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:47.727750 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.727758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.729520 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.729524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.730461 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.730474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.731659 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.731663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.731928 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.732592 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:47.732620 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:47.735320 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:47.738646 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.738652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.738656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.739123 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.739506 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.739513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.739518 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.739666 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.740273 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:47.740277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:47.740421 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 06:24:47.740574 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 06:24:47.740870 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":37}
I20260809 06:24:47.741460 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.725952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.3.137.125:46501)
0809 06:24:47.726620 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:47.726632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:47.726670 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:24:47.726737 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:47.726747 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:47.727049 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:47.727221 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:47.727942 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.729385 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.730617 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.731511 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.731659 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.735568 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:47.735581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:47.735597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:47.735628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:47.738491 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.739261 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.739274 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.739276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.739394 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.739771 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.739779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.739782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.740144 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:47.740369 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:47.740739 (+ 370us) server_negotiation.cc:299] Negotiation successful
0809 06:24:47.740971 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":463,"thread_start_us":289,"threads_started":1}
I20260809 06:24:47.744264 27937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:47.745609 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.745113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59040)
0809 06:24:47.745349 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.745450 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 06:24:47.745909 27882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.747639 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.747305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59054)
0809 06:24:47.747448 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.747532 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 06:24:47.747893 27937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
I20260809 06:24:47.772444 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:46457
--webserver_interface=127.3.137.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:24:47.797245 27937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 1
I20260809 06:24:47.798537 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.798008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59066)
0809 06:24:47.798229 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.798372 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:24:47.799826 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.799435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59068)
0809 06:24:47.799600 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.799692 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 06:24:47.800120 27937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.852479 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } attempt: 5
Aug 09 06:24:47 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:24:47.868033 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.853641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.3.137.125:46501)
0809 06:24:47.853847 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:47.853854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:47.853882 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:24:47.853950 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:47.853956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:47.854053 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:47.854169 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:47.855111 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.855861 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.857131 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.857666 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.857829 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.861948 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:47.861954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:47.861958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:47.862000 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:47.865012 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.865824 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.865842 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.865848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.866001 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:47.866404 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:47.866409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:47.866412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:47.866729 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:47.866906 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:47.867455 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 06:24:47.867696 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:24:47.868036 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.853372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37368)
0809 06:24:47.853632 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.853656 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:47.853818 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:47.854215 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:47.854223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:47.854654 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:47.854887 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.854898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.856025 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.856030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.856954 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:47.856968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:47.857881 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:47.857888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:47.858143 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:47.859011 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:47.859042 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:47.861624 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:47.865202 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.865209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.865213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.865648 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.866128 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:47.866133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:47.866137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:47.866287 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:47.866862 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:47.866868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:47.867110 (+ 242us) client_negotiation.cc:769] Sending connection context
0809 06:24:47.867293 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 06:24:47.867641 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 06:24:47.868564 27937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 2
I20260809 06:24:47.869581 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.869143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59070)
0809 06:24:47.869336 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.869422 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:24:47.869674 27856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:47.870762 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.870396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59080)
0809 06:24:47.870544 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.870612 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 06:24:47.871009 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.870409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59078)
0809 06:24:47.870672 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.870795 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260809 06:24:47.871109 27937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.872095 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.871688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59096)
0809 06:24:47.871858 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.871952 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 06:24:47.872371 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.923641 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 1
I20260809 06:24:47.925026 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.924436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59106)
0809 06:24:47.924676 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.924798 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 06:24:47.926311 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.925892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59108)
0809 06:24:47.926057 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.926150 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 06:24:47.926546 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.962466 27937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 3
I20260809 06:24:47.963934 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.963400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59116)
0809 06:24:47.963628 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.963754 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 06:24:47.965090 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.964618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59128)
0809 06:24:47.964880 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.964964 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260809 06:24:47.965345 27937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:47.994814 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 2
I20260809 06:24:47.996402 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.995679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59142)
0809 06:24:47.995994 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.996150 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:24:47.997658 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:47.997189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59148)
0809 06:24:47.997376 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:47.997483 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 06:24:47.997911 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:48.104209 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 3
I20260809 06:24:48.105686 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.104998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59156)
0809 06:24:48.105257 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.105430 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:24:48.106935 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.106464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59170)
0809 06:24:48.106676 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.106776 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260809 06:24:48.107198 27856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
I20260809 06:24:48.108572 27937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 4
I20260809 06:24:48.109632 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.109169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59172)
0809 06:24:48.109404 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.109495 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:24:48.110697 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.110387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59176)
0809 06:24:48.110522 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.110592 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:24:48.111004 27937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:46457: Network error: Client connection negotiation failed: client connection to 127.3.137.124:46457: connect: Connection refused (error 111)
W20260809 06:24:48.143129 27946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:48.143482 27946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:48.143594 27946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:48.143719 27946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:48.151317 27946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:48.151412 27946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:48.151503 27946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:48.151582 27946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:48.151680 27946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:48.161785 27946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:44511,127.3.137.125:46501,127.3.137.124:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.137.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:48.163126 27946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:48.164423 27946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:48.176420 27951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:48.176453 27952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:48.176497 27954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:48.177412 27946 server_base.cc:1034] running on GCE node
I20260809 06:24:48.178215 27946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:48.179677 27946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:48.180930 27946 hybrid_clock.cc:648] HybridClock initialized: now 1786256688180862 us; error 40 us; skew 500 ppm
I20260809 06:24:48.181263 27946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:48.184484 27946 init.cc:377] Logged in from keytab as kudu/127.3.137.124@KRBTEST.COM (short username kudu)
I20260809 06:24:48.185860 27946 webserver.cc:460] Webserver started at http://127.3.137.124:37455/ using document root <none> and password file <none>
I20260809 06:24:48.186559 27946 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:48.186648 27946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:48.187007 27946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:48.189939 27946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data/instance:
uuid: "082068336b0b4ad5b2077b18f601a99d"
format_stamp: "Formatted at 2026-08-09 06:24:48 on dist-test-slave-2w3w"
server_key: "c54bf8e491a9f4fee07830c51378045f"
server_key_iv: "b8c3aa33fafa0b098ae7cd70adf38d7e"
server_key_version: "encryptionkey@0"
I20260809 06:24:48.190671 27946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal/instance:
uuid: "082068336b0b4ad5b2077b18f601a99d"
format_stamp: "Formatted at 2026-08-09 06:24:48 on dist-test-slave-2w3w"
server_key: "c54bf8e491a9f4fee07830c51378045f"
server_key_iv: "b8c3aa33fafa0b098ae7cd70adf38d7e"
server_key_version: "encryptionkey@0"
I20260809 06:24:48.196101 27946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:24:48.199620 27962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:48.200995 27946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:24:48.201148 27946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "082068336b0b4ad5b2077b18f601a99d"
format_stamp: "Formatted at 2026-08-09 06:24:48 on dist-test-slave-2w3w"
server_key: "c54bf8e491a9f4fee07830c51378045f"
server_key_iv: "b8c3aa33fafa0b098ae7cd70adf38d7e"
server_key_version: "encryptionkey@0"
I20260809 06:24:48.201342 27946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:48.213268 27946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:48.217386 27946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:48.217706 27946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:48.236610 27946 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.124:46457
I20260809 06:24:48.236619 28018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.124:46457 every 8 connection(s)
I20260809 06:24:48.238018 27946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
I20260809 06:24:48.241751 28019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:48.248452 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 27946
I20260809 06:24:48.248669 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786256324325922-3621-0/minicluster-data/master-2/wal/instance
I20260809 06:24:48.249166 3621 external_mini_cluster.cc:1428] Setting key ef61d2cebb83ded4ca521aef39522e75
I20260809 06:24:48.249429 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:48.270625 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.252599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.3.137.126:44511)
0809 06:24:48.254001 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.254005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.254023 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:24:48.254791 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.254797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.254924 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.255095 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.256439 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.257241 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.258936 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.259417 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.259593 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.263390 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.263395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.263398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.263433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.267039 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.267863 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.267872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.267878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.268040 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.268655 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.268661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.268665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.268996 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.269198 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.269972 (+ 774us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.270203 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":1122,"threads_started":1}
I20260809 06:24:48.271811 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.251205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.3.137.124:46457)
0809 06:24:48.252018 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.252028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.252065 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:24:48.254454 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.254480 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.254730 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.254949 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.256669 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.257943 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.259368 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.260193 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.260335 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.264658 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.264672 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.264691 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.264742 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.268249 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.268933 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.268942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.268945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.269080 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.269460 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.269464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.269468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.269822 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.270141 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.270440 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.271049 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260809 06:24:48.271850 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.252554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:44511 (local address 127.0.0.1:59400)
0809 06:24:48.253391 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.253432 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:48.254468 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:48.255186 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:48.255193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:48.255856 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:48.256230 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.256247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.257418 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.257422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.258752 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.258765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.259698 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.259706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.260018 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.260833 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:48.260858 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:48.263129 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:48.267178 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.267187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.267198 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.267743 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.268332 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.268335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.268338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.268523 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.269167 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:48.269175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:48.269556 (+ 381us) client_negotiation.cc:769] Sending connection context
0809 06:24:48.269878 (+ 322us) client_negotiation.cc:240] Negotiation successful
0809 06:24:48.270422 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":46,"thread_start_us":389,"threads_started":1}
I20260809 06:24:48.273502 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:48.284430 27856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 4
I20260809 06:24:48.287844 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.274144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:46501 (local address 127.0.0.1:37380)
0809 06:24:48.274280 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.274297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:48.274418 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:48.274905 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:48.274910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:48.275389 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:48.275538 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.275546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.276533 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.276540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.277547 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.277559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.278271 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.278276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.278482 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.279068 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:48.279085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:48.281270 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:48.285003 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.285008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.285012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.285582 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.286057 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.286061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.286065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.286247 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.286770 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:48.286774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:48.286940 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 06:24:48.287164 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 06:24:48.287487 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 06:24:48.288204 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.274287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.3.137.125:46501)
0809 06:24:48.274510 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.274517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.274536 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:24:48.274599 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.274605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.274682 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.274804 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.275684 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.276396 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.277682 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.278119 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.278262 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.281478 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.281485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.281492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.281538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.284864 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.285726 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.285732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.285736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.285906 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.286359 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.286364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.286368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.286650 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.286806 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.287292 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.287532 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:24:48.289361 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:48.292384 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.275242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.3.137.126:44511)
0809 06:24:48.275415 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.275422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.275464 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:24:48.275658 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.275663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.275742 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.275839 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.277051 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.277850 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.279144 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.279589 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.279722 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.284374 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.284379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.284383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.284417 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.288176 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.289179 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.289184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.289188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.289349 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.289776 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.289782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.289786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.290185 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.290477 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.291751 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.292043 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:48.301683 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.285590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.3.137.124:46457)
0809 06:24:48.285767 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.285773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.285790 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:24:48.285840 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.285844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.285912 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.286008 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.287056 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.288106 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.289868 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.290675 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.290853 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.295020 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.295029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.295034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.295073 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.298620 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.299486 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.299493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.299496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.299662 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.300135 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.300140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.300143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.300417 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.300586 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.301078 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.301327 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260809 06:24:48.302574 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.285115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59194)
0809 06:24:48.285527 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.285548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:48.285664 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:48.286054 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:48.286061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:48.286623 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:48.286869 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.286883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.288222 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.288228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.289685 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.289701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.290855 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.290862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.291129 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.291805 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:48.291825 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:48.294702 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:48.298771 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.298777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.298782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.299343 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.299830 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.299837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.299842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.300025 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.300552 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:48.300558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:48.300744 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 06:24:48.301876 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0809 06:24:48.302173 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256688, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:48.310046 27856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: Bootstrap starting.
I20260809 06:24:48.312031 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.294551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.3.137.125:46501)
0809 06:24:48.294775 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.294793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.294858 (+ 65us) server_negotiation.cc:407] Connection header received
0809 06:24:48.295101 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.295109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.295204 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.295327 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.297035 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.298222 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.300010 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.300687 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.300815 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.306121 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.306126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.306129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.306161 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.309222 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.309928 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.309932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.309934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.310069 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.310543 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.310548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.310551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.310828 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.311153 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.311469 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.311695 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:24:48.314419 27856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:48.315354 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.290169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:59204)
0809 06:24:48.290432 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.290449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:48.290558 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:48.299826 (+ 9268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:48.299831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:48.300368 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:48.300565 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.300576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.302375 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.302380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.303421 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.303433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.304511 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.304517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.304714 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.305364 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:48.305388 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:48.308171 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:48.312360 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.312368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.312373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.312854 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.313297 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.313304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.313308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.313472 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.314153 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:48.314160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:48.314366 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 06:24:48.314579 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 06:24:48.314950 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1}
I20260809 06:24:48.315809 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.290130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.3.137.124:46457)
0809 06:24:48.299497 (+ 9367us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.299502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.299524 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:24:48.299570 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.299576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.299652 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.299777 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.300708 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.302239 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.303866 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.304370 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.304501 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.308404 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.308414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.308419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.308468 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.312223 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.312993 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.312997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.313001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.313149 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.313588 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.313596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.313600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.314012 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.314155 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.314689 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.315403 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":435,"server-negotiator.queue_time_us":9205,"thread_start_us":8986,"threads_started":1}
I20260809 06:24:48.315914 27856 log.cc:824] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:48.317444 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:48.319321 27856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: No bootstrap required, opened a new log
I20260809 06:24:48.321231 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: Bootstrap starting.
I20260809 06:24:48.320130 27831 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a3ef59dbb460449095a8c1fb44df06ef"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412
I20260809 06:24:48.324569 28019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:48.325830 28019 log.cc:824] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:48.329023 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: No bootstrap required, opened a new log
I20260809 06:24:48.328786 27856 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.329197 27856 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ef59dbb460449095a8c1fb44df06ef, State: Initialized, Role: FOLLOWER
I20260809 06:24:48.329748 27856 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.330703 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.331128 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.331925 27856 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:48.332082 27831 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 06:24:48.332269 27831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:48.335196 27831 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.337427 27831 leader_election.cc:288] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [CANDIDATE]: Term 1 election: Requested vote from peers a114833b6bcb43db87a99047a02b5d44 (127.3.137.125:46501), 082068336b0b4ad5b2077b18f601a99d (127.3.137.124:46457)
I20260809 06:24:48.341786 28019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.342344 28019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082068336b0b4ad5b2077b18f601a99d, State: Initialized, Role: FOLLOWER
I20260809 06:24:48.343047 28019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.350126 28019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:48.350212 27991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3ef59dbb460449095a8c1fb44df06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "082068336b0b4ad5b2077b18f601a99d"
I20260809 06:24:48.350869 27991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:48.347506 28037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.352162 28037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.355522 27991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ef59dbb460449095a8c1fb44df06ef in term 1.
I20260809 06:24:48.356465 27805 leader_election.cc:302] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082068336b0b4ad5b2077b18f601a99d, a3ef59dbb460449095a8c1fb44df06ef; no voters:
I20260809 06:24:48.357216 28028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:48.374204 28028 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [term 1 LEADER]: Becoming Leader. State: Replica: a3ef59dbb460449095a8c1fb44df06ef, State: Running, Role: LEADER
I20260809 06:24:48.375031 28028 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
W20260809 06:24:48.377700 27809 tablet.cc:2369] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:48.387686 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ef59dbb460449095a8c1fb44df06ef. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.388353 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:48.399478 28062 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:24:48.406585 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.362343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.3.137.125:46501)
0809 06:24:48.362574 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.362584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.373093 (+ 10509us) server_negotiation.cc:407] Connection header received
0809 06:24:48.373651 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.373660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.373763 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.373849 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.376218 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.377356 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.380937 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.381621 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.381736 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.397438 (+ 15702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.397448 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.397456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.397530 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.402056 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.403048 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.403055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.403059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.403235 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.403777 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.403785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.403789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.404214 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.404483 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.405935 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.406212 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:24:48.407563 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.365991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.3.137.124:46457)
0809 06:24:48.366179 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.366186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.376135 (+ 9949us) server_negotiation.cc:407] Connection header received
0809 06:24:48.376693 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.376701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.376818 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.376907 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.384292 (+ 7385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.385855 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.390203 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.391166 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.391323 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.398235 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.398255 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.398263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.398322 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.402444 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.403893 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.403898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.403902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.404150 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.404961 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.404970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.404973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.405619 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.406006 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.406921 (+ 915us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.407210 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 06:24:48.410276 27937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } attempt: 5
Aug 09 06:24:48 dist-test-slave-2w3w krb5kdc[27773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
W20260809 06:24:48.428535 28056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:24:48.428730 28056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:24:48.417199 27946 master_runner.cc:186] Error getting master registration for 127.3.137.125:46501: OK, instance_id { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" instance_seqno: 1786256687680963 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:24:48.430065 27946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:24:48.430145 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.411024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:46457 (local address 127.0.0.1:56060)
0809 06:24:48.412580 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:48.412608 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:48.412754 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:48.413108 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:48.413126 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:48.413587 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:48.413744 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.413763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.415713 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.415718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.417079 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:48.417094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.418210 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.418218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:48.418479 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.419255 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:48.419280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:48.422313 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:48.426823 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.426830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.426834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.427418 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.427892 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:48.427900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:48.427904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:48.428075 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:48.428961 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:48.428968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:48.429130 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 06:24:48.429294 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 06:24:48.429606 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376}
I20260809 06:24:48.431391 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.411046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.3.137.124:46457)
0809 06:24:48.411291 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.411296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.412694 (+ 1398us) server_negotiation.cc:407] Connection header received
0809 06:24:48.412881 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.412887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.412963 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.413066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.413904 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.415550 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.417296 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.418020 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.422507 (+ 4487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.422790 (+ 283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.422796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.422801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.422833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.426669 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.427591 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.427597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.427600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.427749 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.428272 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.428277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.428281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.428674 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.430627 (+ 1953us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.430767 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.431039 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:24:48.432004 28057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:48.434149 28057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:48.436342 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: Bootstrap starting.
I20260809 06:24:48.440110 27937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:48.441473 27937 log.cc:824] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:48.444942 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: No bootstrap required, opened a new log
I20260809 06:24:48.446282 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:48.368014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.3.137.126:44511)
0809 06:24:48.368240 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.368248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.382182 (+ 13934us) server_negotiation.cc:407] Connection header received
0809 06:24:48.382454 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.382459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.382535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.382611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.397126 (+ 14515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.412194 (+ 15068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.428993 (+ 16799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:48.430023 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:48.430162 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:48.438020 (+ 7858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:48.438026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:48.438030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:48.438082 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:48.441962 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.442800 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.442806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.442810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.442997 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:48.443565 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:48.443570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:48.443575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:48.444297 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:48.444824 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:48.445153 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 06:24:48.445563 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:24:48.392694 27787 master_runner.cc:186] Error getting master registration for 127.3.137.125:46501: OK, instance_id { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" instance_seqno: 1786256687680963 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:24:48.449820 27787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:24:48.455765 27937 raft_consensus.cc:348] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.456238 27937 raft_consensus.cc:729] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a114833b6bcb43db87a99047a02b5d44, State: Initialized, Role: FOLLOWER
I20260809 06:24:48.456862 27937 consensus_queue.cc:260] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } }
I20260809 06:24:48.457078 27991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [term 1 FOLLOWER]: Refusing update from remote peer a3ef59dbb460449095a8c1fb44df06ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:48.458097 28044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:48.458436 28065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.459164 28065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.461041 27937 sys_catalog.cc:564] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:48.461160 27910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3ef59dbb460449095a8c1fb44df06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a114833b6bcb43db87a99047a02b5d44"
I20260809 06:24:48.461623 27910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:48.465337 27910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ef59dbb460449095a8c1fb44df06ef in term 1.
I20260809 06:24:48.466264 27909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [term 1 FOLLOWER]: Refusing update from remote peer a3ef59dbb460449095a8c1fb44df06ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:48.467201 28044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:24:48.469769 27968 tablet.cc:2369] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:48.476516 28037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ef59dbb460449095a8c1fb44df06ef. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.476867 28037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.479959 28064 mvcc.cc:205] Tried to move back new op lower bound from 7316507395910049792 to 7316507395590967296. Current Snapshot: MvccSnapshot[applied={T|T < 7316507395910049792}]
I20260809 06:24:48.480860 28070 mvcc.cc:205] Tried to move back new op lower bound from 7316507395910049792 to 7316507395590967296. Current Snapshot: MvccSnapshot[applied={T|T < 7316507395910049792}]
I20260809 06:24:48.482081 28037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.482407 28037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.491578 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.491865 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:48.492717 28057 catalog_manager.cc:1347] Generated new cluster ID: febc5f75cf6542949748690399effe2f
I20260809 06:24:48.492830 28057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:48.495913 28065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ef59dbb460449095a8c1fb44df06ef. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.496308 28065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.499760 28073 mvcc.cc:205] Tried to move back new op lower bound from 7316507395910049792 to 7316507395590967296. Current Snapshot: MvccSnapshot[applied={T|T < 7316507395910049792}]
I20260809 06:24:48.503273 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.504189 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:48.507619 28065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3ef59dbb460449095a8c1fb44df06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ef59dbb460449095a8c1fb44df06ef" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 44511 } } peers { permanent_uuid: "a114833b6bcb43db87a99047a02b5d44" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 46501 } } peers { permanent_uuid: "082068336b0b4ad5b2077b18f601a99d" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 46457 } } }
I20260809 06:24:48.507871 28065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:48.514142 28085 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:24:48.516709 28057 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:48.518927 28057 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:48.533524 28057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3ef59dbb460449095a8c1fb44df06ef: Generated new TSK 0
I20260809 06:24:48.534525 28057 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:24:48.627421 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.613833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.3.137.125:46501)
0809 06:24:48.614022 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.614029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.624978 (+ 10949us) server_negotiation.cc:407] Connection header received
0809 06:24:48.625567 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.625574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.625667 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.625749 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.627154 (+ 1405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52902: BlockingRecv error: recv got EOF from 127.0.0.1:52902 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 06:24:48.630414 28025 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.613362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.3.137.124:46457)
0809 06:24:48.613611 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.613621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.628513 (+ 14892us) server_negotiation.cc:407] Connection header received
0809 06:24:48.628810 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.628816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.628905 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.628980 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.630135 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56068: BlockingRecv error: recv got EOF from 127.0.0.1:56068 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 06:24:48.631639 28024 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.613147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.3.137.126:44511)
0809 06:24:48.613668 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.613676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.629695 (+ 16019us) server_negotiation.cc:407] Connection header received
0809 06:24:48.629988 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.629993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.630081 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.630151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.631382 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59794: BlockingRecv error: recv got EOF from 127.0.0.1:59794 (error 108)
Metrics: {"server-negotiator.queue_time_us":350}
W20260809 06:24:48.638537 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.636637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.3.137.125:46501)
0809 06:24:48.636908 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.636914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.637120 (+ 206us) server_negotiation.cc:407] Connection header received
0809 06:24:48.637302 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.637307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.637466 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.637621 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.638284 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52914: BlockingRecv error: recv got EOF from 127.0.0.1:52914 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260809 06:24:48.644702 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.643071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.3.137.125:46501)
0809 06:24:48.643249 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.643255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.643393 (+ 138us) server_negotiation.cc:407] Connection header received
0809 06:24:48.643576 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.643581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.643669 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.643773 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.644503 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52922: BlockingRecv error: recv got EOF from 127.0.0.1:52922 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 06:24:48.651819 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.649606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.3.137.125:46501)
0809 06:24:48.649849 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.649858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.650340 (+ 482us) server_negotiation.cc:407] Connection header received
0809 06:24:48.650464 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.650469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.650550 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.650654 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.651557 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52926: BlockingRecv error: recv got EOF from 127.0.0.1:52926 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 06:24:48.661116 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.659064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.3.137.125:46501)
0809 06:24:48.659335 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.659341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.659657 (+ 316us) server_negotiation.cc:407] Connection header received
0809 06:24:48.659857 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.659861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.659928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.660009 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.660880 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52934: BlockingRecv error: recv got EOF from 127.0.0.1:52934 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260809 06:24:48.675806 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.673182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.3.137.125:46501)
0809 06:24:48.673360 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.673365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.673982 (+ 617us) server_negotiation.cc:407] Connection header received
0809 06:24:48.674242 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.674247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.674327 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.674427 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.675497 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52948: BlockingRecv error: recv got EOF from 127.0.0.1:52948 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 06:24:48.698251 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.695833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.3.137.125:46501)
0809 06:24:48.696081 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.696091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.696675 (+ 584us) server_negotiation.cc:407] Connection header received
0809 06:24:48.696917 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.696921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.697016 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.697131 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.697985 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52960: BlockingRecv error: recv got EOF from 127.0.0.1:52960 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 06:24:48.736912 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.734670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.3.137.125:46501)
0809 06:24:48.734938 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.734945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.735235 (+ 290us) server_negotiation.cc:407] Connection header received
0809 06:24:48.735386 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.735392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.735481 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.735632 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.736624 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52976: BlockingRecv error: recv got EOF from 127.0.0.1:52976 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
W20260809 06:24:48.807022 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.804626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.3.137.125:46501)
0809 06:24:48.804891 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.804898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.805296 (+ 398us) server_negotiation.cc:407] Connection header received
0809 06:24:48.805548 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.805552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.805625 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.805716 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.806764 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52986: BlockingRecv error: recv got EOF from 127.0.0.1:52986 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 06:24:48.942844 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:48.940797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.3.137.125:46501)
0809 06:24:48.941064 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:48.941070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:48.941396 (+ 326us) server_negotiation.cc:407] Connection header received
0809 06:24:48.941551 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:48.941555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:48.941629 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:48.941718 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:48.942573 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53002: BlockingRecv error: recv got EOF from 127.0.0.1:53002 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 06:24:49.206779 27943 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:49.204957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.3.137.125:46501)
0809 06:24:49.205202 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.205207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.205383 (+ 176us) server_negotiation.cc:407] Connection header received
0809 06:24:49.205565 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.205569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.205645 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.205743 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:49.206542 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53008: BlockingRecv error: recv got EOF from 127.0.0.1:53008 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 06:24:49.404318 28062 catalog_manager.cc:1259] Loaded cluster ID: febc5f75cf6542949748690399effe2f
I20260809 06:24:49.404454 28062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: loading cluster ID for follower catalog manager: success
I20260809 06:24:49.407434 28062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: acquiring CA information for follower catalog manager: success
I20260809 06:24:49.409301 28062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 082068336b0b4ad5b2077b18f601a99d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:24:49.520121 28085 catalog_manager.cc:1259] Loaded cluster ID: febc5f75cf6542949748690399effe2f
I20260809 06:24:49.520265 28085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: loading cluster ID for follower catalog manager: success
I20260809 06:24:49.524348 28085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: acquiring CA information for follower catalog manager: success
I20260809 06:24:49.527076 28085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a114833b6bcb43db87a99047a02b5d44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:24:49.755164 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:49.733242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.3.137.125:46501)
0809 06:24:49.745712 (+ 12470us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.745720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.746630 (+ 910us) server_negotiation.cc:407] Connection header received
0809 06:24:49.746882 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.746888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.747170 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.747368 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:49.748042 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.749444 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.751799 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.752590 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.752733 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:49.753472 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:49.754335 (+ 863us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:49.754504 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:49.754644 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 06:24:49.754754 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12292,"thread_start_us":12477,"threads_started":1}
I20260809 06:24:49.788882 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:49.773225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.3.137.124:46457)
0809 06:24:49.781080 (+ 7855us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.781086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.781475 (+ 389us) server_negotiation.cc:407] Connection header received
0809 06:24:49.781831 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.781835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.781987 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.782151 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:49.782836 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.783793 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.785929 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.786384 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.786493 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:49.787237 (+ 744us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:49.788010 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:49.788130 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:49.788347 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 06:24:49.788440 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7699,"thread_start_us":7511,"threads_started":1}
I20260809 06:24:49.832078 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:49.797226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.3.137.126:44511)
0809 06:24:49.820783 (+ 23557us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.820790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.821880 (+ 1090us) server_negotiation.cc:407] Connection header received
0809 06:24:49.822044 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.822051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.822318 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.822532 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:49.823639 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.824843 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.827313 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.828004 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.828157 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:49.830142 (+ 1985us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:49.830661 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:49.830779 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:49.831557 (+ 778us) server_negotiation.cc:299] Negotiation successful
0809 06:24:49.831661 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23373,"thread_start_us":23584,"threads_started":1}
I20260809 06:24:49.838891 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:49.797610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.3.137.125:46501)
0809 06:24:49.798034 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.798044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.829354 (+ 31310us) server_negotiation.cc:407] Connection header received
0809 06:24:49.829482 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.829492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.829638 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.829770 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:49.830821 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.832552 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.836403 (+ 3851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.837100 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.837258 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:49.837947 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:49.838115 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:49.838208 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:49.838413 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 06:24:49.838505 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260809 06:24:49.840806 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27787
I20260809 06:24:49.841307 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:49.797757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.3.137.124:46457)
0809 06:24:49.798238 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:49.798246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:49.831454 (+ 33208us) server_negotiation.cc:407] Connection header received
0809 06:24:49.833091 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:49.833098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:49.833202 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:49.833277 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:49.834292 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.835484 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.838052 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:49.838929 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:49.839063 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:49.840369 (+ 1306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:49.840516 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:49.840596 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:49.840884 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 06:24:49.840989 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260809 06:24:49.841997 27851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:49.961634 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27787
I20260809 06:24:49.986316 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27863
I20260809 06:24:49.987546 27931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:50.099711 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27863
I20260809 06:24:50.123003 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 27946
I20260809 06:24:50.124213 28012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:50.233771 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 27946
2026-08-09T06:24:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5121 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:24:50 dist-test-slave-2w3w krb5kdc[28179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:24:50 dist-test-slave-2w3w krb5kdc[28179](info): set up 2 sockets
Aug 09 06:24:50 dist-test-slave-2w3w krb5kdc[28179](info): commencing operation
krb5kdc: starting...
W20260809 06:24:51.430094 3621 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/test-user.keytab.
Aug 09 06:24:51 dist-test-slave-2w3w krb5kdc[28179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.137.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
Entry for principal kudu/127.3.137.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab.
I20260809 06:24:51.784948 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39843
--webserver_interface=127.3.137.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:24:52.151196 28192 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:52.151554 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:52.151629 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:52.151714 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:52.159363 28192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:52.159480 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:52.159541 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:52.159597 28192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:52.159665 28192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:52.170297 28192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.126.keytab
--principal=kudu/127.3.137.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.126:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.137.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:52.171837 28192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:52.173247 28192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:52.186160 28200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:52.186232 28197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:52.186240 28198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:52.187325 28192 server_base.cc:1034] running on GCE node
I20260809 06:24:52.188504 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:52.190147 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:52.191406 28192 hybrid_clock.cc:648] HybridClock initialized: now 1786256692191336 us; error 47 us; skew 500 ppm
I20260809 06:24:52.191778 28192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:52.195149 28192 init.cc:377] Logged in from keytab as kudu/127.3.137.126@KRBTEST.COM (short username kudu)
I20260809 06:24:52.196666 28192 webserver.cc:460] Webserver started at http://127.3.137.126:39319/ using document root <none> and password file <none>
I20260809 06:24:52.197634 28192 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:52.197933 28192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:52.198318 28192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:52.201670 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data/instance:
uuid: "569323f49af849e5afc4948697687fb7"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "d77a4a61d5f3b277c5a956049b07e145"
server_key_iv: "fa8d8ca7f59d8d5b9850310ffbe5bb1c"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.202539 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance:
uuid: "569323f49af849e5afc4948697687fb7"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "d77a4a61d5f3b277c5a956049b07e145"
server_key_iv: "fa8d8ca7f59d8d5b9850310ffbe5bb1c"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.208335 28192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 06:24:52.212214 28207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:52.213737 28192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:24:52.213981 28192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal
uuid: "569323f49af849e5afc4948697687fb7"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "d77a4a61d5f3b277c5a956049b07e145"
server_key_iv: "fa8d8ca7f59d8d5b9850310ffbe5bb1c"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.214232 28192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:52.231180 28192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:52.235457 28192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:52.235749 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:52.255286 28192 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.126:39843
I20260809 06:24:52.255309 28261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.126:39843 every 8 connection(s)
I20260809 06:24:52.256767 28192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/data/info.pb
I20260809 06:24:52.260666 28262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:52.260914 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 28192
I20260809 06:24:52.261106 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-0/wal/instance
I20260809 06:24:52.261569 3621 external_mini_cluster.cc:1428] Setting key fd50604bffd9985def837c2eb12dcb6f
I20260809 06:24:52.267380 28262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:52.290076 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.270520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.3.137.126:39843)
0809 06:24:52.272203 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:52.272211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:52.272262 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:24:52.273820 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:52.273826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:52.274072 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:52.274280 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:52.275704 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.276950 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.278416 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.279002 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.279211 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.282569 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:52.282577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:52.282590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:52.282618 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:52.285605 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.286275 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.286285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.286289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.286438 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.286838 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.286847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.286850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.287408 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:52.287621 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:52.288168 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 06:24:52.288576 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":1455,"thread_start_us":1231,"threads_started":1}
I20260809 06:24:52.290094 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.270519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39843 (local address 127.0.0.1:37248)
0809 06:24:52.271400 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.271449 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:52.273446 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:52.274295 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:52.274302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:52.275103 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:52.275485 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.275504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.277102 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.277107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.278266 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.278278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.279253 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.279259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.279449 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.280133 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:52.280155 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:52.282335 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:52.285729 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.285742 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.285755 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.286149 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.286546 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.286552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.286556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.286723 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.287543 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:52.287555 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:52.287775 (+ 220us) client_negotiation.cc:769] Sending connection context
0809 06:24:52.288032 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 06:24:52.288694 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":159,"thread_start_us":380,"threads_started":1}
WARNING: no policy specified for kudu/127.3.137.125@KRBTEST.COM; defaulting to no policy
I20260809 06:24:52.293658 28262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:52.294917 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.294401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48414)
0809 06:24:52.294562 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.294676 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 06:24:52.295205 28211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
I20260809 06:24:52.296831 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.296382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48430)
0809 06:24:52.296526 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.296601 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.125@KRBTEST.COM" created.
W20260809 06:24:52.297045 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:40233: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
Entry for principal kudu/127.3.137.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab.
I20260809 06:24:52.339903 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:40233
--webserver_interface=127.3.137.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:24:52.346467 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } attempt: 1
I20260809 06:24:52.348004 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.347349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48436)
0809 06:24:52.347714 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.347870 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260809 06:24:52.349040 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.348668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48452)
0809 06:24:52.348834 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.348906 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 06:24:52.349241 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:40233: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
I20260809 06:24:52.417527 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } attempt: 2
I20260809 06:24:52.419344 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.418613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48462)
0809 06:24:52.418952 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.419110 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 06:24:52.420617 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.420166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48464)
0809 06:24:52.420349 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.420441 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:24:52.420931 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:40233: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
I20260809 06:24:52.512246 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } attempt: 3
I20260809 06:24:52.513542 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.513004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48472)
0809 06:24:52.513251 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.513368 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:24:52.514899 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.514343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48486)
0809 06:24:52.514546 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.514661 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 06:24:52.515275 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:40233: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
I20260809 06:24:52.658586 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } attempt: 4
I20260809 06:24:52.660141 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.659431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48496)
0809 06:24:52.659814 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.659942 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260809 06:24:52.661402 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.660907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48502)
0809 06:24:52.661086 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.661206 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 06:24:52.661679 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.125:40233: Network error: Client connection negotiation failed: client connection to 127.3.137.125:40233: connect: Connection refused (error 111)
W20260809 06:24:52.701624 28270 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:52.702044 28270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:52.702188 28270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:52.702332 28270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:52.710021 28270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:52.710146 28270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:52.710209 28270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:52.710320 28270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:52.710417 28270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:52.721483 28270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.125.keytab
--principal=kudu/127.3.137.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.125:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.137.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:52.723029 28270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:52.724543 28270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:52.737943 28275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:52.737975 28277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:52.738015 28279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:52.738899 28270 server_base.cc:1034] running on GCE node
I20260809 06:24:52.739869 28270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:52.741537 28270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:52.742781 28270 hybrid_clock.cc:648] HybridClock initialized: now 1786256692742710 us; error 45 us; skew 500 ppm
I20260809 06:24:52.743227 28270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:52.746902 28270 init.cc:377] Logged in from keytab as kudu/127.3.137.125@KRBTEST.COM (short username kudu)
I20260809 06:24:52.748618 28270 webserver.cc:460] Webserver started at http://127.3.137.125:38765/ using document root <none> and password file <none>
I20260809 06:24:52.749338 28270 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:52.749442 28270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:52.749709 28270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:52.752825 28270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data/instance:
uuid: "895cf364ee8648b0a1a758495c43a2a0"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "2acb2fd0cf45788eb791606cc813dce6"
server_key_iv: "d1f21906f16c335712cbdacf7be93e2a"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.753629 28270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal/instance:
uuid: "895cf364ee8648b0a1a758495c43a2a0"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "2acb2fd0cf45788eb791606cc813dce6"
server_key_iv: "d1f21906f16c335712cbdacf7be93e2a"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.759166 28270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:24:52.762720 28286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:52.764132 28270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:24:52.764319 28270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal
uuid: "895cf364ee8648b0a1a758495c43a2a0"
format_stamp: "Formatted at 2026-08-09 06:24:52 on dist-test-slave-2w3w"
server_key: "2acb2fd0cf45788eb791606cc813dce6"
server_key_iv: "d1f21906f16c335712cbdacf7be93e2a"
server_key_version: "encryptionkey@0"
I20260809 06:24:52.764546 28270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:52.787906 28270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:52.792424 28270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:52.792709 28270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:52.812518 28270 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.125:40233
I20260809 06:24:52.812536 28338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.125:40233 every 8 connection(s)
I20260809 06:24:52.813973 28270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/data/info.pb
I20260809 06:24:52.815277 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 28270
I20260809 06:24:52.815477 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-1/wal/instance
I20260809 06:24:52.816077 3621 external_mini_cluster.cc:1428] Setting key 00e105fae56f52a49dbb4a46e239f6cc
I20260809 06:24:52.818889 28339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:52.829138 28339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.137.124@KRBTEST.COM; defaulting to no policy
I20260809 06:24:52.852509 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.832789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.3.137.126:39843)
0809 06:24:52.833354 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:52.833358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:52.834483 (+ 1125us) server_negotiation.cc:407] Connection header received
0809 06:24:52.836489 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:52.836495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:52.836588 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:52.836713 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:52.838380 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.839175 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.840735 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.841157 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.841297 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.845550 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:52.845556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:52.845559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:52.845613 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:52.848775 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.849692 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.849703 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.849705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.849846 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.850336 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.850343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.850346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.850654 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:52.850829 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:52.851791 (+ 962us) server_negotiation.cc:299] Negotiation successful
0809 06:24:52.852057 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":304,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.137.124@KRBTEST.COM" created.
I20260809 06:24:52.853737 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.833513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39843 (local address 127.0.0.1:37262)
0809 06:24:52.834302 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.834354 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:52.836205 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:52.836984 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:52.836993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:52.837554 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:52.838142 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.838160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.839343 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.839348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.840563 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.840576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.841398 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.841405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.841673 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.842436 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:52.842461 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:52.845235 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:52.848971 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.848982 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.849009 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.849543 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.849969 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.849974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.849978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.850208 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.850794 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:52.850802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:52.851329 (+ 527us) client_negotiation.cc:769] Sending connection context
0809 06:24:52.851686 (+ 357us) client_negotiation.cc:240] Negotiation successful
0809 06:24:52.852346 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":326,"threads_started":1}
I20260809 06:24:52.855882 28339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:24:52.874131 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.856651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48512)
0809 06:24:52.856881 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.856907 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:52.857050 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:52.858150 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:52.858157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:52.858708 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:52.858934 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.858949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.860365 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.860373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.861839 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.861857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.862724 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.862732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.863025 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.863704 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:52.863733 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:52.866693 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:52.871458 (+ 4765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.871462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.871465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.871886 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.872345 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.872349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.872351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.872507 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.873349 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:52.873354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:52.873479 (+ 125us) client_negotiation.cc:769] Sending connection context
0809 06:24:52.873598 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 06:24:52.873819 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 06:24:52.874660 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.856978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.3.137.125:40233)
0809 06:24:52.857678 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:52.857684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:52.857712 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:24:52.857760 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:52.857767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:52.857975 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:52.858096 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:52.859144 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.860213 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.861990 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.862560 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.862669 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.867047 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:52.867066 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:52.867083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:52.867123 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:52.871258 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.872041 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.872053 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.872058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.872219 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.872630 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.872638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.872644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.873201 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:52.873437 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:52.873789 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 06:24:52.874101 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":393,"threads_started":1}
I20260809 06:24:52.877348 28339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:52.878753 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.878160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53166)
0809 06:24:52.878393 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.878550 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260809 06:24:52.879123 28288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:52.880906 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.880443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53176)
0809 06:24:52.880642 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.880755 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 06:24:52.881182 28339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
Entry for principal kudu/127.3.137.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab.
I20260809 06:24:52.899188 3621 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
/tmp/dist-test-task6Z6xHV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:40155
--webserver_interface=127.3.137.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.137.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:24:52.930557 28339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 1
I20260809 06:24:52.932067 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.931448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53182)
0809 06:24:52.931703 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.931848 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 06:24:52.933238 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.932750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53186)
0809 06:24:52.932952 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.933059 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 06:24:52.933490 28339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:52.960991 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } attempt: 5
Aug 09 06:24:52 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:24:52.977690 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.961798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48526)
0809 06:24:52.962119 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.962148 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:52.962396 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:52.962866 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:52.962876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:52.963611 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:52.963923 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.963943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.965152 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.965162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.966545 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:52.966564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.967558 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.967565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:52.967805 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.968567 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:52.968592 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:52.971166 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:52.975067 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.975071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.975074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.975510 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.975954 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:52.975958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:52.975960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:52.976091 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:52.976802 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:52.976806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:52.976975 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 06:24:52.977097 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 06:24:52.977429 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 06:24:52.977823 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.961986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.3.137.125:40233)
0809 06:24:52.962180 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:52.962184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:52.962226 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:24:52.962574 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:52.962578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:52.962671 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:52.962759 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:52.964159 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.964964 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.966730 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:52.967389 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:52.967496 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:52.971381 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:52.971387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:52.971392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:52.971436 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:52.974840 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.975642 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.975647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.975651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.975825 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:52.976206 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:52.976211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:52.976214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:52.976615 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:52.976776 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:52.977262 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 06:24:52.977510 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:24:52.978946 28262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:52.980213 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.979720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53190)
0809 06:24:52.979920 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.980025 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:24:52.981388 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:52.980940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53200)
0809 06:24:52.981104 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:52.981247 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:24:52.981750 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.001747 28339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 2
I20260809 06:24:53.003101 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.002515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53204)
0809 06:24:53.002743 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.002862 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 06:24:53.004233 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.003833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53216)
0809 06:24:53.003990 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.004078 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 06:24:53.004504 28339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.033025 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 1
I20260809 06:24:53.034466 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.033875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53222)
0809 06:24:53.034116 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.034245 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 06:24:53.035682 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.035300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53228)
0809 06:24:53.035481 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.035565 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260809 06:24:53.035916 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.095749 28339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 3
I20260809 06:24:53.097194 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.096504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53234)
0809 06:24:53.096794 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.096941 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 06:24:53.098520 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.097949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53236)
0809 06:24:53.098248 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.098373 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260809 06:24:53.098767 28339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.104144 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 2
I20260809 06:24:53.105424 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.104896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53242)
0809 06:24:53.105138 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.105229 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:24:53.106446 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.106105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53256)
0809 06:24:53.106253 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.106347 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:24:53.106700 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.212950 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 3
I20260809 06:24:53.214155 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.213672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53266)
0809 06:24:53.213883 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.213977 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:24:53.215430 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.214898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53280)
0809 06:24:53.215154 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.215261 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260809 06:24:53.215690 28262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
I20260809 06:24:53.242022 28339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 4
I20260809 06:24:53.243376 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.242758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53292)
0809 06:24:53.243049 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.243156 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 06:24:53.244462 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.244046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53302)
0809 06:24:53.244197 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.244323 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:24:53.244704 28339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.137.124:40155: Network error: Client connection negotiation failed: client connection to 127.3.137.124:40155: connect: Connection refused (error 111)
W20260809 06:24:53.266875 28349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:24:53.267269 28349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:24:53.267349 28349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:24:53.267433 28349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:24:53.274943 28349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:24:53.275110 28349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:24:53.275174 28349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:24:53.275226 28349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:24:53.275290 28349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:24:53.285545 28349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.137.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.137.126:39843,127.3.137.125:40233,127.3.137.124:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/krb5kdc/kudu_127.3.137.124.keytab
--principal=kudu/127.3.137.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.137.124:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.137.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:36:50 UTC on ccycloud.quasar-ooxuuz.root.comops.site
build id 4575857
ASAN enabled
I20260809 06:24:53.286892 28349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:24:53.288427 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:24:53.301261 28356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:24:53.301297 28358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:53.302476 28349 server_base.cc:1034] running on GCE node
W20260809 06:24:53.302700 28355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:24:53.303726 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:24:53.305236 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:24:53.306497 28349 hybrid_clock.cc:648] HybridClock initialized: now 1786256693306439 us; error 40 us; skew 500 ppm
I20260809 06:24:53.306886 28349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:24:53.310179 28349 init.cc:377] Logged in from keytab as kudu/127.3.137.124@KRBTEST.COM (short username kudu)
I20260809 06:24:53.311709 28349 webserver.cc:460] Webserver started at http://127.3.137.124:40535/ using document root <none> and password file <none>
I20260809 06:24:53.312378 28349 fs_manager.cc:359] Metadata directory not provided
I20260809 06:24:53.312466 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:24:53.312748 28349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:24:53.315855 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data/instance:
uuid: "e1354b466cb14fc293152ea40e7f859a"
format_stamp: "Formatted at 2026-08-09 06:24:53 on dist-test-slave-2w3w"
server_key: "604367ad2551314821dba089f99438c8"
server_key_iv: "45cfceb1c097860a356102bce3c41d75"
server_key_version: "encryptionkey@0"
I20260809 06:24:53.316578 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal/instance:
uuid: "e1354b466cb14fc293152ea40e7f859a"
format_stamp: "Formatted at 2026-08-09 06:24:53 on dist-test-slave-2w3w"
server_key: "604367ad2551314821dba089f99438c8"
server_key_iv: "45cfceb1c097860a356102bce3c41d75"
server_key_version: "encryptionkey@0"
I20260809 06:24:53.321812 28349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:24:53.325297 28366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:24:53.326683 28349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:24:53.326910 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data,/tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal
uuid: "e1354b466cb14fc293152ea40e7f859a"
format_stamp: "Formatted at 2026-08-09 06:24:53 on dist-test-slave-2w3w"
server_key: "604367ad2551314821dba089f99438c8"
server_key_iv: "45cfceb1c097860a356102bce3c41d75"
server_key_version: "encryptionkey@0"
I20260809 06:24:53.327169 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:24:53.342085 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:24:53.346298 28349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:24:53.346536 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:24:53.365962 28349 rpc_server.cc:307] RPC server started. Bound to: 127.3.137.124:40155
I20260809 06:24:53.365993 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.137.124:40155 every 8 connection(s)
I20260809 06:24:53.367448 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/data/info.pb
I20260809 06:24:53.371362 28423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:24:53.375339 3621 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu as pid 28349
I20260809 06:24:53.375530 3621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Z6xHV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786256324325922-3621-0/minicluster-data/master-2/wal/instance
I20260809 06:24:53.376044 3621 external_mini_cluster.cc:1428] Setting key 4a694d870f7b1b620bf18aa3d3be12e2
I20260809 06:24:53.378935 28423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:53.392947 28262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 4
I20260809 06:24:53.402781 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.383212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.3.137.126:39843)
0809 06:24:53.383782 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.383788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.384356 (+ 568us) server_negotiation.cc:407] Connection header received
0809 06:24:53.384570 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.384576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.384656 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.384771 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.386040 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.386742 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.389092 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.389502 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.389603 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.393960 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.393967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.393971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.394010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.398636 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.399791 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.399827 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.399831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.400036 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.400476 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.400481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.400485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.400829 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.400980 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.402119 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.402339 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
I20260809 06:24:53.403190 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.378172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.3.137.124:40155)
0809 06:24:53.379106 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.379118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.379164 (+ 46us) server_negotiation.cc:407] Connection header received
0809 06:24:53.381027 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.381053 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.381526 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.381865 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.383705 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.385395 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.386866 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.387782 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.388059 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.393238 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.393253 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.393271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.393324 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.397461 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.398212 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.398225 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.398237 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.398411 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.398943 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.398947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.398949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.399949 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.400309 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.401187 (+ 878us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.401614 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":374,"threads_started":1}
I20260809 06:24:53.403404 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.383194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.126:39843 (local address 127.0.0.1:37278)
0809 06:24:53.384200 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.384251 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:53.384418 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:53.384845 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:53.384849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:53.385525 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:53.385872 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.385885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.387056 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.387060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.388760 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.388795 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.389753 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.389761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.390008 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.390680 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:53.390720 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:53.393678 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:53.398766 (+ 5088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.398777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.398785 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.399636 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.400164 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.400168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.400171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.400339 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.401055 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:53.401065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:53.401313 (+ 248us) client_negotiation.cc:769] Sending connection context
0809 06:24:53.401526 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 06:24:53.401874 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":572,"threads_started":1}
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.126@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:53.405480 28423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:53.411856 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.393683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53316)
0809 06:24:53.394338 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.394369 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:53.394497 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:53.395725 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:53.395736 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:53.396355 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:53.396604 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.396618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.398743 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.398749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.399964 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.399978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.400996 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.401003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.401241 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.401869 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:53.401891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:53.404772 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:53.409192 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.409197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.409201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.409613 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.410021 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.410026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.410029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.410173 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.410760 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:53.410765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:53.410923 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 06:24:53.411126 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 06:24:53.411432 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260809 06:24:53.411875 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.393967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.3.137.124:40155)
0809 06:24:53.395358 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.395362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.395384 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:24:53.395423 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.395426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.395527 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.395640 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.397097 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.398586 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.400153 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.400817 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.401558 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.405017 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.405022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.405025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.405058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.409059 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.409732 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.409739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.409743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.409896 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.410292 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.410299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.410305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.410598 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.410771 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.411221 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.411457 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1276,"thread_start_us":1106,"threads_started":1}
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.126@KRBTEST.COM
I20260809 06:24:53.419286 28262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: Bootstrap starting.
I20260809 06:24:53.423738 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.406365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.3.137.125:40233)
0809 06:24:53.406594 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.406601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.406655 (+ 54us) server_negotiation.cc:407] Connection header received
0809 06:24:53.406883 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.406890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.407003 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.407115 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.408131 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.409199 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.411071 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.411744 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.411890 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.416147 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.416154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.416158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.416203 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.420060 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.421084 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.421090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.421094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.421284 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.421846 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.421853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.421857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.422270 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.422469 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.423038 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.423308 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:24:53.424173 28262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:53.424822 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.406220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.125:40233 (local address 127.0.0.1:48540)
0809 06:24:53.406570 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.406591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:53.406718 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:53.407141 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:53.407146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:53.407688 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:53.407938 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.407955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.409419 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.409426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.410900 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.410915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.412119 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.412125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.412359 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.412998 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:53.413018 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:53.415899 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:53.420275 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.420282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.420288 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.420892 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.421490 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.421497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.421503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.421702 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.422439 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:53.422444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:53.422641 (+ 197us) client_negotiation.cc:769] Sending connection context
0809 06:24:53.424057 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0809 06:24:53.424427 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260809 06:24:53.425993 28423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:24:53.426023 28262 log.cc:824] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:53.426257 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.407884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.3.137.126:39843)
0809 06:24:53.408065 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.408071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.408192 (+ 121us) server_negotiation.cc:407] Connection header received
0809 06:24:53.408345 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.408350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.408441 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.408554 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.410194 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.411259 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.413755 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.414234 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.414382 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.419162 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.419176 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.419179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.419219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.422569 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.423465 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.423472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.423476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.423627 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.424119 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.424126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.424129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.424537 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.424980 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.425493 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.425757 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.124@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:53.436543 28262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: No bootstrap required, opened a new log
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.137.125@KRBTEST.COM
I20260809 06:24:53.443397 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.426820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53328)
0809 06:24:53.427060 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.427081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:53.427194 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:53.427741 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:53.427745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:53.428291 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:53.428503 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.428517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.429877 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.429948 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.431350 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.431363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.432108 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.432113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.432304 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.433007 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:53.433041 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:53.436711 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:53.440801 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.440808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.440812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.441211 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.441702 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.441705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.441708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.441844 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.442433 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:53.442444 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:53.442622 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 06:24:53.442780 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 06:24:53.443136 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:24:53.444146 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.427016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.3.137.124:40155)
0809 06:24:53.427168 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.427173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.427191 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:24:53.427384 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.427389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.427480 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.427553 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.428660 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.429702 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.431495 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.431952 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.432174 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.436968 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.436976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.436982 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.437030 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.440661 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.441404 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.441412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.441415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.441575 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.441978 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.441983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.441986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.442294 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.442493 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.443751 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.443943 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:24:53.447387 28262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.447850 28262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569323f49af849e5afc4948697687fb7, State: Initialized, Role: FOLLOWER
I20260809 06:24:53.447975 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.428392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.3.137.125:40233)
0809 06:24:53.428619 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.428628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.428653 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:24:53.428817 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.428826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.428955 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.429092 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.430862 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.432515 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.433844 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.434293 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.434455 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.440529 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.440537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.440542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.440592 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.444572 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.445626 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.445634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.445638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.445830 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.446326 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.446331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.446336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.446699 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.447044 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.447390 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.447644 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:24:53.448442 28262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.449625 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.450045 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.450915 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: Bootstrap starting.
I20260809 06:24:53.451009 28262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:53.453438 3621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:24:53.456300 28236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "569323f49af849e5afc4948697687fb7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37294
I20260809 06:24:53.456626 28236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 06:24:53.456821 28236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:53.457774 28423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:53.458895 28423 log.cc:824] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:53.459293 28236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.460856 28236 leader_election.cc:288] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 895cf364ee8648b0a1a758495c43a2a0 (127.3.137.125:40233), e1354b466cb14fc293152ea40e7f859a (127.3.137.124:40155)
I20260809 06:24:53.468573 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: No bootstrap required, opened a new log
I20260809 06:24:53.467017 28192 master_runner.cc:186] Error getting master registration for 127.3.137.125:40233: OK, instance_id { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" instance_seqno: 1786256692807112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260809 06:24:53.479064 28445 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:24:53.479249 28445 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:24:53.479422 28192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:24:53.484901 28423 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.485566 28423 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1354b466cb14fc293152ea40e7f859a, State: Initialized, Role: FOLLOWER
I20260809 06:24:53.486327 28423 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
W20260809 06:24:53.488828 28214 tablet.cc:2369] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:53.488126 28451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.489389 28451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.490056 28423 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:53.490139 28396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "569323f49af849e5afc4948697687fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1354b466cb14fc293152ea40e7f859a"
I20260809 06:24:53.490688 28396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:53.494478 28396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569323f49af849e5afc4948697687fb7 in term 1.
I20260809 06:24:53.495432 28209 leader_election.cc:302] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569323f49af849e5afc4948697687fb7, e1354b466cb14fc293152ea40e7f859a; no voters:
I20260809 06:24:53.496022 28433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:24:53.496788 28433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 569323f49af849e5afc4948697687fb7, State: Running, Role: LEADER
I20260809 06:24:53.497423 28433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.516885 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569323f49af849e5afc4948697687fb7. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.517403 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:53.541556 28461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:24:53.543915 28339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } attempt: 5
I20260809 06:24:53.546319 28461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:24:53.553314 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.499031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.3.137.124:40155)
0809 06:24:53.499254 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.499262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.510542 (+ 11280us) server_negotiation.cc:407] Connection header received
0809 06:24:53.510741 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.510750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.510903 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.511061 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.515689 (+ 4628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.525438 (+ 9749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.527759 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.528523 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.528644 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.541977 (+ 13333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.541986 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.541990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.542036 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.546416 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.548498 (+ 2082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.548505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.548509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.548694 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.550175 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.550181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.550185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.550585 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.550910 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.552611 (+ 1701us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.552909 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 06:24:53 dist-test-slave-2w3w krb5kdc[28179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786256692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.137.125@KRBTEST.COM for kudu/127.3.137.124@KRBTEST.COM
I20260809 06:24:53.564291 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.524590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.3.137.125:40233)
0809 06:24:53.524839 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.524848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.532200 (+ 7352us) server_negotiation.cc:407] Connection header received
0809 06:24:53.532875 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.532884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.533061 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.533208 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.536092 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.537694 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.540408 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.541433 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.541565 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.549120 (+ 7555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.549127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.549132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.549175 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.553228 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.557129 (+ 3901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.557139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.557145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.557396 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.559784 (+ 2388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.559793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.559799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.560290 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.560627 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.561513 (+ 886us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.563738 (+ 2225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 06:24:53.567312 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.498473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.3.137.126:39843)
0809 06:24:53.498696 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.498706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.524138 (+ 25432us) server_negotiation.cc:407] Connection header received
0809 06:24:53.533791 (+ 9653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.533799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.533909 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.533997 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.537770 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.539038 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.546553 (+ 7515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.547284 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.547405 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.559466 (+ 12061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.559473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.559478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.559564 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.563548 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.564721 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.564727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.564731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.564899 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.565376 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.565383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.565386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.565778 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.566060 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.566661 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.566904 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:24:53.581869 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.544581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.137.124:40155 (local address 127.0.0.1:53346)
0809 06:24:53.544996 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 06:24:53.545017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:24:53.545125 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:24:53.546020 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:24:53.546026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:24:53.546519 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:24:53.546752 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.546767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.548263 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.548270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.549860 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:24:53.549877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.551148 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.551155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:24:53.551481 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.552164 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:24:53.552187 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:24:53.563395 (+ 11208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:24:53.573738 (+ 10343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.573746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.573753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.574539 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.575320 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:24:53.575327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:24:53.575334 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:24:53.575644 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:24:53.580295 (+ 4651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:24:53.580304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:24:53.580561 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 06:24:53.580777 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 06:24:53.581207 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260809 06:24:53.583174 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:53.545371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.3.137.124:40155)
0809 06:24:53.545548 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.545554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.545573 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:24:53.545629 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.545634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.545705 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.545804 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.546945 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.548083 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.550063 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:53.550756 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:53.550919 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:53.566313 (+ 15394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:24:53.566323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:24:53.566331 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:24:53.566400 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:24:53.571280 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.574754 (+ 3474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.574763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.574769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.575047 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:24:53.575820 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:24:53.575828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:24:53.575834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:24:53.576389 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:24:53.576558 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:53.582268 (+ 5710us) server_negotiation.cc:299] Negotiation successful
0809 06:24:53.582672 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 06:24:53.598662 28372 tablet.cc:2369] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:24:53.602794 28396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [term 1 FOLLOWER]: Refusing update from remote peer 569323f49af849e5afc4948697687fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:53.606712 28456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:53.607514 28339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: Bootstrap starting.
I20260809 06:24:53.619518 28339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: Neither blocks nor log segments found. Creating new log.
I20260809 06:24:53.620939 28339 log.cc:824] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: Log is configured to *not* fsync() on all Append() calls
I20260809 06:24:53.626384 28339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: No bootstrap required, opened a new log
I20260809 06:24:53.596278 28349 master_runner.cc:186] Error getting master registration for 127.3.137.125:40233: OK, instance_id { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" instance_seqno: 1786256692807112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:24:53.630568 28349 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 06:24:53.633492 28470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:24:53.633637 28470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:24:53.643245 28451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569323f49af849e5afc4948697687fb7. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.643579 28451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.647615 28339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.648028 28465 mvcc.cc:205] Tried to move back new op lower bound from 7316507416982380544 to 7316507416570834944. Current Snapshot: MvccSnapshot[applied={T|T < 7316507416982380544}]
I20260809 06:24:53.648092 28339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895cf364ee8648b0a1a758495c43a2a0, State: Initialized, Role: FOLLOWER
I20260809 06:24:53.648675 28339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } }
I20260809 06:24:53.655982 28476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.656442 28476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.658522 28339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:24:53.658583 28313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "569323f49af849e5afc4948697687fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "895cf364ee8648b0a1a758495c43a2a0"
I20260809 06:24:53.658975 28312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:24:53.664832 28312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [term 1 FOLLOWER]: Refusing update from remote peer 569323f49af849e5afc4948697687fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:24:53.669543 28313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569323f49af849e5afc4948697687fb7 in term 1.
I20260809 06:24:53.673544 28433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:24:53.688349 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.688710 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:53.690725 28461 catalog_manager.cc:1347] Generated new cluster ID: 9655e05265854ed29519df7e105baaa2
I20260809 06:24:53.690869 28461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:24:53.704298 28451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.704716 28451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.728156 28484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.728631 28484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.728991 28476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569323f49af849e5afc4948697687fb7. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.729234 28476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:24:53.733152 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "569323f49af849e5afc4948697687fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569323f49af849e5afc4948697687fb7" member_type: VOTER last_known_addr { host: "127.3.137.126" port: 39843 } } peers { permanent_uuid: "895cf364ee8648b0a1a758495c43a2a0" member_type: VOTER last_known_addr { host: "127.3.137.125" port: 40233 } } peers { permanent_uuid: "e1354b466cb14fc293152ea40e7f859a" member_type: VOTER last_known_addr { host: "127.3.137.124" port: 40155 } } }
I20260809 06:24:53.733554 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:24:53.738078 28494 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:24:53.746053 28461 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:24:53.748569 28461 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:24:53.774205 28461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569323f49af849e5afc4948697687fb7: Generated new TSK 0
I20260809 06:24:53.775272 28461 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:24:53.879436 28427 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.876753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.3.137.124:40155)
0809 06:24:53.877034 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.877041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.877406 (+ 365us) server_negotiation.cc:407] Connection header received
0809 06:24:53.877708 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.877714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.877806 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.877884 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.879155 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53348: BlockingRecv error: recv got EOF from 127.0.0.1:53348 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 06:24:53.881846 28428 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.877465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.3.137.126:39843)
0809 06:24:53.877688 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.877694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.880251 (+ 2557us) server_negotiation.cc:407] Connection header received
0809 06:24:53.880518 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.880525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.880608 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.880680 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.881609 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37312: BlockingRecv error: recv got EOF from 127.0.0.1:37312 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 06:24:53.885170 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.883389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.3.137.125:40233)
0809 06:24:53.883600 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.883606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.883689 (+ 83us) server_negotiation.cc:407] Connection header received
0809 06:24:53.883834 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.883840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.883927 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.884030 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.884937 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48556: BlockingRecv error: recv got EOF from 127.0.0.1:48556 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 06:24:53.899470 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.897092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.3.137.125:40233)
0809 06:24:53.897296 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.897305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.897812 (+ 507us) server_negotiation.cc:407] Connection header received
0809 06:24:53.898070 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.898076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.898205 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.898313 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.899183 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48560: BlockingRecv error: recv got EOF from 127.0.0.1:48560 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 06:24:53.909127 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.903615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.3.137.125:40233)
0809 06:24:53.903780 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.903786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.907578 (+ 3792us) server_negotiation.cc:407] Connection header received
0809 06:24:53.907775 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.907783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.907887 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.908005 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.908890 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48564: BlockingRecv error: recv got EOF from 127.0.0.1:48564 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 06:24:53.916487 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.914264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.3.137.125:40233)
0809 06:24:53.914442 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.914447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.914955 (+ 508us) server_negotiation.cc:407] Connection header received
0809 06:24:53.915171 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.915177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.915320 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.915431 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.916239 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48576: BlockingRecv error: recv got EOF from 127.0.0.1:48576 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 06:24:53.935063 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.932448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.3.137.125:40233)
0809 06:24:53.932700 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.932707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.933321 (+ 614us) server_negotiation.cc:407] Connection header received
0809 06:24:53.933625 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.933632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.933749 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.933873 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.934794 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48584: BlockingRecv error: recv got EOF from 127.0.0.1:48584 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260809 06:24:53.949537 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.946685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.3.137.125:40233)
0809 06:24:53.946880 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.946886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.947570 (+ 684us) server_negotiation.cc:407] Connection header received
0809 06:24:53.947784 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.947791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.947910 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.948034 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.949282 (+ 1248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48586: BlockingRecv error: recv got EOF from 127.0.0.1:48586 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 06:24:53.971946 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:53.969338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.3.137.125:40233)
0809 06:24:53.969535 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:53.969542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:53.970125 (+ 583us) server_negotiation.cc:407] Connection header received
0809 06:24:53.970349 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:53.970356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:53.970462 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:53.970692 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:53.971687 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48596: BlockingRecv error: recv got EOF from 127.0.0.1:48596 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 06:24:54.010511 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:54.008003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.3.137.125:40233)
0809 06:24:54.008216 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:54.008223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:54.008807 (+ 584us) server_negotiation.cc:407] Connection header received
0809 06:24:54.009043 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:54.009051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:54.009157 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:54.009285 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:54.010226 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48608: BlockingRecv error: recv got EOF from 127.0.0.1:48608 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 06:24:54.080533 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:54.078179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.3.137.125:40233)
0809 06:24:54.078447 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:54.078453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:54.078800 (+ 347us) server_negotiation.cc:407] Connection header received
0809 06:24:54.079144 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:54.079149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:54.079231 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:54.079329 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:54.080297 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48616: BlockingRecv error: recv got EOF from 127.0.0.1:48616 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260809 06:24:54.218953 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:54.216832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.3.137.125:40233)
0809 06:24:54.217137 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:54.217144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:54.217439 (+ 295us) server_negotiation.cc:407] Connection header received
0809 06:24:54.217626 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:54.217631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:54.217704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:54.217793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:54.218678 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48624: BlockingRecv error: recv got EOF from 127.0.0.1:48624 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260809 06:24:54.482388 28345 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:24:54.480494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.3.137.125:40233)
0809 06:24:54.480764 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:54.480772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:54.480983 (+ 211us) server_negotiation.cc:407] Connection header received
0809 06:24:54.481199 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:54.481203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:54.481279 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:54.481377 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:24:54.482149 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48626: BlockingRecv error: recv got EOF from 127.0.0.1:48626 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 06:24:54.635914 28470 catalog_manager.cc:1259] Loaded cluster ID: 9655e05265854ed29519df7e105baaa2
I20260809 06:24:54.636044 28470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: loading cluster ID for follower catalog manager: success
I20260809 06:24:54.638525 28470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: acquiring CA information for follower catalog manager: success
I20260809 06:24:54.640373 28470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1354b466cb14fc293152ea40e7f859a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:24:54.742851 28494 catalog_manager.cc:1259] Loaded cluster ID: 9655e05265854ed29519df7e105baaa2
I20260809 06:24:54.743264 28494 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: loading cluster ID for follower catalog manager: success
I20260809 06:24:54.745942 28494 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: acquiring CA information for follower catalog manager: success
I20260809 06:24:54.747908 28494 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 895cf364ee8648b0a1a758495c43a2a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:24:55.006942 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:54.998755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.3.137.125:40233)
0809 06:24:54.999879 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:54.999884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:54.999909 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:24:54.999977 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:54.999981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:55.000179 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:55.000292 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:55.000938 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.001966 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.004157 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.004636 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.004746 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:55.005451 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:55.006105 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:55.006221 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:55.006425 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 06:24:55.006508 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":854,"threads_started":1}
I20260809 06:24:55.020365 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:55.014176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.3.137.124:40155)
0809 06:24:55.014901 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:55.014907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:55.014929 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:24:55.015065 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:55.015069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:55.015226 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:55.015434 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:55.015851 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.016597 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.017944 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.018435 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.018579 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:55.019131 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:55.019724 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:55.019880 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:55.019960 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 06:24:55.020055 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":422,"threads_started":1}
I20260809 06:24:55.041970 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:55.027927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.3.137.124:40155)
0809 06:24:55.028183 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:55.028195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:55.033512 (+ 5317us) server_negotiation.cc:407] Connection header received
0809 06:24:55.033806 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:55.033812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:55.033924 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:55.033994 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:55.035012 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.036319 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.039200 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.039910 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.040015 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:55.041025 (+ 1010us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:55.041183 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:55.041255 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:55.041506 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 06:24:55.041600 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:24:55.042508 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:55.027174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.3.137.126:39843)
0809 06:24:55.031926 (+ 4752us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:55.031933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:55.031983 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:24:55.032152 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:55.032158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:55.032416 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:55.032653 (+ 237us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:55.033537 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.035115 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.039142 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.039892 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.040029 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:55.041085 (+ 1056us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:55.041619 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:55.041702 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:55.042048 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 06:24:55.042148 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4597,"thread_start_us":4419,"threads_started":1}
I20260809 06:24:55.046728 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:24:55.033629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.3.137.125:40233)
0809 06:24:55.033822 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:24:55.033828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:24:55.038990 (+ 5162us) server_negotiation.cc:407] Connection header received
0809 06:24:55.039226 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:24:55.039232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:24:55.039342 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:24:55.039407 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:24:55.040225 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.041349 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.044081 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:24:55.044786 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:24:55.044897 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:24:55.045620 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:24:55.045791 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:24:55.045909 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:24:55.046162 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 06:24:55.046294 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:24:55.052496 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 28192
I20260809 06:24:55.055387 28257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:55.203603 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 28192
I20260809 06:24:55.228799 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 28270
I20260809 06:24:55.229943 28334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:55.339583 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 28270
I20260809 06:24:55.364027 3621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu pid 28349
I20260809 06:24:55.365116 28418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:24:55.473579 3621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Z6xHV/build/asan/bin/kudu with pid 28349
2026-08-09T06:24:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5239 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10361 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (371072 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST